Pipeline Laravel Package

Technical Evaluation

Architecture fit. sanmai/pipeline is a framework-agnostic, generator-based data pipeline library. It ships no service provider, facade, config, or Laravel-specific glue, so its "architecture fit" is entirely a question of whether it fills a gap in our data-processing layer rather than how it plugs into Laravel's container. The honest starting position is that Laravel already has a first-party incumbent: Illuminate\Support\LazyCollection, which covers lazy map/filter/chunk/zip/flatMap and is backed by Eloquent's cursor() and chunkById(). The case for this package therefore rests on a narrow set of capabilities LazyCollection does not offer: reservoir() (weighted reservoir sampling), runningVariance() / finalVariance() (Welford's online algorithm), the zip() + unpack() transposition idiom, chunkBy() with variable sizes, and a mutable pipeline object designed to be handed between services so each layer can append its own stage. That last property — every stage-adding method returns the same instance — is architecturally interesting for rule-engine and plugin-style processing, and is simultaneously the biggest source of surprise for developers expecting a fluent immutable API.

Integration feasibility. Low effort, low coupling. It is a single Composer dependency with no transitive framework assumptions; adoption is composer require sanmai/pipeline and requires no bootstrapping. The constraint is PHP version: the current release requires PHP 8.2+, which maps cleanly onto Laravel 11 and 12 but excludes Laravel 10 and earlier unless we pin an older, less feature-complete release. Because there is no framework coupling, a reversal is cheap — the blast radius of removal is the set of call sites, not the application skeleton.

Technical risk. Three risks dominate, and none of them are runtime-quality risks:

1. Incumbent substitution. The package's headline value ("the missing pipe operator") is largely already delivered by LazyCollection at zero marginal dependency cost, with first-party support and deeper Eloquent integration. Opportunity cost of introducing a second, overlapping idiom into the codebase is real.
2. Maintainer concentration. 74 stars, 0 listed dependents, and a single-maintainer profile imply a bus factor of one. The library's own quality signals are strong (100% line coverage, 100% mutation score with Infection, Psalm + PHPStan including generic/type-inference tests, multi-version CI matrix, a DocumentationTest that enforces README/API sync), which suggests the maintainer is disciplined — but discipline is not continuity.
3. Semantic footguns. Pipelines are non-rewindable; re-iterating a consumed pipeline produces "Cannot traverse an already closed generator." iterator_to_array() silently drops duplicate keys, so toList()/toAssoc() are mandatory. Non-generator callbacks into map() force the eager array path and can spike memory. Mutability means a pipeline held in a singleton will accumulate stages across requests under Octane.

The "no exceptions thrown, ever" design is a genuine asset for stream processing of untrusted input: the library never manufactures failure, so any exception in a pipeline is definitively ours.

Key questions.

• Which of our workloads cannot be expressed in LazyCollection, and does any of them actually need reservoir sampling, online variance, or zip/unpack transposition?
• Are we prepared to own a fork if upstream goes dark, given the package is small and Apache-2.0 (fork-friendly)?
• Do we have engineering fluency in the lazy-vs-eager duality, or will we need to codify rules (always yield, never return, from a seeding callback)?
• Will pipelines be confined to console commands and queued jobs, or will they leak into request-scoped services where Octane state reuse is a hazard?
• Is the release cadence credible? The listed latest release date is in the future relative to typical audit windows, which should be reconciled against Packagist before we plan upgrades around it.
• Do we need per-stage observability (timing, cardinality, drop rates)? The library offers no hooks; we would wrap stages ourselves via tap().

Integration Approach

Stack fit. The package is a pure library, so "integration" means establishing an internal idiom, not wiring. It fits naturally in console commands, queued jobs, and import/export services — long-running, memory-sensitive, single-consumer work. It fits poorly in controllers and Blade rendering, where re-iteration of a one-shot iterator and debugging via dd()/dump() are common and will surface the non-rewindability caveat as apparent bugs.

PHP 8.2+ alignment is the gating compatibility fact:

Under Octane, every pipeline must be constructed per request/job; never bind a Pipeline\Standard into the container as a singleton, because the mutable-instance design will accumulate stages across requests.

Migration path. Treat this as a supplement, not a replacement. The pragmatic sequence is:

1. Identify call sites currently using collect()->chunk(...) chains or array_map/array_filter compositions over large result sets where memory is the observed constraint.
2. Rewrite the handful of those that need stream semantics using Pipeline\take() over an Eloquent cursor(), or over a generator that pages with chunkById().
3. Replace only the LazyCollection usages where a specific missing operator (reservoir, variance, zip/unpack) is the blocker — do not migrate working LazyCollection code for stylistic uniformity.
4. Confine early adopters to new code paths only. No big-bang rewrite.

Compatibility. Coexistence with Collections and LazyCollection is straightforward because the pipeline accepts any Traversable and implements IteratorAggregate and Countable; it can be seeded from a collection and drained into one. Practical bridging rules to standardize internally:

• Seed from generators, not arrays, whenever the source is large or the callback is expensive.
• Drain with toList() / toAssoc(), never iterator_to_array().
• Use cursor() where partial consumption or resumption is needed, rather than relying on rewind behavior.
• Prefer runningCount() over count() in streamed contexts, since count() is a terminal operation.

Sequencing. A four-step rollout: (a) a one-week spike proving the two or three specific operators LazyCollection lacks against a real backlog dataset; (b) an internal adapter or thin wrapper class so call sites depend on our surface, not Pipeline\Standard directly — this is the single most valuable risk mitigation given the dependency profile; (c) production use in one bounded, non-customer-facing batch job with memory instrumentation before/after; (d) a written idiom guide and code-review checklist covering laziness, keys, and rewind rules. Gate step (c) on the spike demonstrating a measurable memory or expressiveness win; if it doesn't, the correct outcome is to standardize on LazyCollection and stop.

Operational Impact

Maintenance. The dependency carries no operational surface — no migrations, no config, no scheduled tasks, no published assets. Ongoing cost is (i) tracking upstream releases against our PHP floor, and (ii) absorbing the bus-factor-one risk. Because the package is one primary class plus small helpers under Apache-2.0, a fork or vendored copy is a cheap contingency; I would budget it as a contingency rather than assume it. The 0-dependents figure should be read as "no ecosystem safety net," not "unusable" — it means we are early adopters and should behave accordingly, pinning to a specific version and reviewing diffs on upgrade rather than floating.

Support. Documentation is above average for a package this size: ReadTheDocs with a quickstart, full API reference, cookbook, and best-practices section, plus an explicit AGENTS.md describing internals (three-state internal storage, array shortcut vs. generator path, arity-wrapping in tap()/each()). That internal documentation materially lowers the cost of debugging and of forking. What does not exist is a Laravel-specific support channel, a maintainer SLA, or a community large enough to expect timely answers to edge-case questions. Internal ownership must be assigned; there is no external escalation path.

Scaling. Per-stage memory is effectively constant for generator-backed pipelines, which is the whole point — this is the right tool for row-by-row transforms over datasets that exceed memory_limit. CPU cost is the generator indirection overhead, typically small relative to I/O but measurable in tight numeric loops. Scaling is strictly vertical and single-threaded: there is no parallelism, no backpressure, and no batching primitive beyond chunk()/chunkBy(). For database work, the pipeline is only as efficient as its seed — pairing it with cursor() (unbuffered) or chunkById() (paged) determines the actual memory profile; a pipeline seeded from ->get() gains nothing.

Failure modes. Ranked by likelihood:

• Closed-generator reuse — iterating a pipeline twice, or passing a consumed pipeline into a second consumer, throws a PHP Exception. Mitigation: cursor(), and never store pipelines in shared scope.
• Silent key collision — iterator_to_array() discards duplicate keys with no warning, so the "missing rows" bug is data-dependent and hard to reproduce. Mitigation: ban iterator_to_array() in review; use toList().
• Accidental eagerness — a map() callback that returns a scalar instead of yielding flips the pipeline onto the array path and can exhaust memory on a large seed. This is a code-review-detectable defect with production-only symptoms.
• Stage leakage under Octane — a container-shared pipeline accumulates stages across requests; symptoms are duplicated side effects (double writes via tap()), not errors.
• Unbounded terminal operations — count(), toList(), fold() against an infinite or very large generator will hang or OOM before any error surfaces. Mitigation: never call terminal operations without a slice()/peek() bound.
• Upstream abandonment — manifests as an inability to support a future PHP major. Mitigation: adapter layer plus fork contingency.

Note that user-supplied callbacks are not protected: the library throws nothing itself, but exceptions from our closures propagate normally. Observability must therefore be added by us, via tap() for sampling and metrics and wrapper callbacks for timing; the library provides no telemetry hooks.

Ramp-up. For developers already comfortable with collect() chains, the mental model transfers in under a day: method names mirror array functions (cast≈array_map, select≈array_filter, fold≈array_reduce, flatten, chunk, zip). The non-obvious parts — mutable-same-instance returns, best-effort key preservation, the eager array shortcut, non-rewindability — take a sprint of real usage and should be captured as a short internal style guide and review checklist rather than left to oral tradition. Expect the first production incident to come from one of those four semantics, not from the library's core logic; the pre-emptive guardrails (banned iterator_to_array(), "always yield" rule, no shared pipeline instances) are cheap and should be in place before the first deployment.