Json Stream Laravel Package

Technical Evaluation

Architecture fit. clue/json-stream is a push-based, incremental JSON parser built on ReactPHP's stream abstractions (react/stream, evenement). It attaches to a ReadableStreamInterface and emits a data event per decoded JSON document, supporting both concatenated JSON ({...}{...}) and newline-delimited JSON. That is a fundamentally asynchronous programming model, while Laravel is request-scoped and synchronous. The package therefore fits only in the long-running fringes of a Laravel application: artisan commands, queue workers, Octane workers, or a dedicated ingest process. It does not belong on the HTTP request path unless the team is already comfortable running an event loop inside a controller.

Integration feasibility. Install is trivial (composer require clue/json-stream), MIT-licensed, and the codebase is small enough (~a parser plus a stream decoder) to vendor or fork if needed. The real gating item is platform currency: the last release was 2020-12-19, and the package was never CI-tested against PHP 8.1–8.3. Since Laravel 10/11 requires PHP 8.1+, we must verify installability and runtime behavior on our target PHP version before committing. Transitive dependencies are light (react/stream, evenement/evenement), but they introduce ReactPHP into the dependency graph, which is a durable architectural commitment, not a drive-by utility.

Technical risk.

• Maintenance/abandonment: 24 stars, single maintainer, no release in four-plus years. Low activity is partially offset by the tiny, stable scope and MIT license (fork is cheap).
• Paradigm mismatch: no Laravel service provider, no facades, no framework idioms. Every integration is bespoke glue.
• Buffering semantics: the parser must hold one complete JSON document in memory before emitting it. NDJSON with very large individual records (multi-MB blobs) will spike memory just like json_decode on that record.
• Correctness under chunk boundaries: the core value proposition is correct handling of splits mid-string, mid-escape, mid-unicode. We should not take this on faith — a conformance suite with adversarial fixtures (1-byte chunks, split \uXXXX, BOM, CRLF, whitespace between documents) is mandatory.
• No backpressure: the producer pushes; a slow consumer (Eloquent writes) silently accumulates decoded documents in memory.

Key questions.

1. What is the actual source of the stream — local file, S3 object, HTTP response body, or socket — and is it push-capable or only pull-capable?
2. Is ReactPHP already in our stack anywhere, or would this be its first introduction?
3. What is the largest single JSON document we must tolerate, and what is our memory ceiling per worker?
4. Do we need resumability/checkpointing after a malformed record, or is full-batch failure acceptable?
5. Has anyone benchmarked halaxa/json-machine (synchronous generator-based streaming, Laravel-friendly, far larger community) against this use case? If not, why is an async parser the default answer?

Integration Approach

Stack fit. Best suited to Laravel Octane or standalone long-running workers where an event loop already exists or can be owned. For classic PHP-FPM deployments, it runs fine inside an artisan command or a queued job that calls Loop::run(), but the loop must be short-lived and exception-safe so it never leaks into the next job.

Recommended pattern — adopt behind a port. Do not let clue/json-stream types bleed into domain code. Define an internal contract, e.g. App\Contracts\JsonStreamReader, with a single method returning an iterable/generator of decoded documents plus a progress/offset reporter. Provide two adapters:

• JsonMachineReader (synchronous, generator-based, zero new dependencies) as the default.
• ClueStreamReader (ReactPHP) for sources that are genuinely push-based or already async.

This keeps the vendor decision reversible and makes the eventual migration a one-line binding change.

Compatibility details to nail down in the spike.

• composer install on PHP 8.2/8.3 with Laravel 10/11 — check for composer.json platform constraints and deprecation notices at runtime.
• Transport adapters: plain file via React\Stream\ReadableResourceStream; S3 via Storage::readStream() piped into a readable stream; gzip via PHP's zlib.inflate stream filter rather than post-hoc decompression.
• Whitespace/BOM/CRLF tolerance and behavior on malformed input (exception type, offset reporting).

Sequencing.

1. Spike (2–3 days): install in a throwaway Laravel 11 app, parse a 1 GB NDJSON fixture, record throughput, peak memory, and failure behavior.
2. Conformance suite: adversarial chunk-boundary fixtures; treat any parser divergence as a blocker.
3. Port + adapters: introduce JsonStreamReader; implement the json-machine adapter first, clue adapter second.
4. Pilot: migrate one non-critical pipeline (e.g. a nightly vendor export ingest) with feature-flagged adapter selection.
5. Measure and decide: if clue wins on throughput/memory materially, standardize; if not, retire the dependency and keep json-machine. Pin the version and record the fork path in the ADR either way.

Operational Impact

Maintenance. Treat this as a "vendor-and-own" dependency: no upstream roadmap, no security response guarantee. Budget a named owner, pin an exact version, and keep a short fork playbook (the code is small enough that a fork is a half-day, not a project). Re-evaluate annually against json-machine's trajectory.

Support. Community-only, MIT, no SLA. Escalation is us. This is acceptable for a batch pipeline, harder to justify for a customer-facing path.

Scaling. Single-threaded and push-based; effective throughput is bounded by the slowest consumer, typically Eloquent inserts. Plan to batch writes (e.g. 500-record chunks with insert() or upsert()) inside the data callback, and to partition large inputs by file or byte range so work parallelizes across workers. Memory is the primary scaling constraint: high-water mark scales with the largest decoded document plus any un-consumed queue, so instrument memory_get_peak_usage() per run and alert on regressions.

Failure modes to design for.

• Malformed record mid-stream: parser throws and the run aborts. Emit the byte offset / record index in the exception and persist a checkpoint so the job can resume rather than restart.
• Memory blowup on large documents: enforce a max-document-size guard before decoding; reject or quarantine oversized records.
• Slow consumer / no backpressure: pause the source stream or, more simply, cap the pipeline at a concurrency of one and monitor heap.
• Stream/network errors: distinguish transient (retry with backoff from last checkpoint) from fatal (dead-letter the batch).
• Event-loop leakage under Octane/queue workers: ensure all streams, timers, and loop references are torn down in a finally block; a leaked loop is a slow-motion memory leak across requests.
• PHP version drift: pin and re-run the conformance suite on every PHP major upgrade.

Ramp-up. A mid-to-senior PHP engineer needs roughly one to two days to become productive, most of which is the ReactPHP mental model (event emitters, stream lifecycle, Loop::run() boundaries) rather than the parser itself. Document the "never call this on the request path" rule explicitly, because that is the most likely and most expensive misuse.

Verdict. Conditional go. Recommended only where the source is genuinely push-based or ReactPHP is already present; otherwise the synchronous, far-better-supported halaxa/json-machine should be the default, with this package evaluated in a bake-off during the spike. The deciding evidence is a measured throughput/memory comparison on a representative 1 GB fixture, not the star count or the description.