ecourty/text-chunker
ecourty/text-chunker is a PHP library for splitting long text into smaller, configurable chunks. It helps prepare content for LLM prompts, embeddings, pagination, SMS, and APIs while keeping chunks readable and within limits.
Architecture fit. ecourty/text-chunker is a framework-agnostic PHP library, not a Laravel package. It exposes a small fluent facade (TextChunker) around three pluggable contracts — ReaderInterface, ChunkingStrategyInterface, ChunkPostProcessorInterface — which maps cleanly onto Laravel's service container and DI conventions. The strategy/pipeline design is idiomatic and extensible: adding a domain-specific chunker is a single class implementing one method. The Generator-based streaming model is the strongest architectural signal for our use case. Ingestion pipelines that process large documents (PDFs, transcripts, scraped HTML) must not buffer full payloads into memory per queue worker; a Generator<Chunk> that yields incrementally is exactly the shape we want. The immutable Chunk value object is trivially testable and safe to pass across pipeline boundaries.
Integration feasibility. High. The library is Composer-installable, MIT-licensed, and has zero framework coupling, so it will not fight Laravel's autoloader, container, or queue system. The main gap is that there is no first-party Laravel service provider, config file, or facade — meaning we own a thin wrapper layer (roughly 1–2 dev-days) to bind strategies/readers into the container, publish configuration, and expose a stable interface to our ingestion jobs. That wrapper is also where we isolate ourselves from upstream API churn, which is desirable regardless.
Technical risk. The dominant risk is not technical — it is project maturity. Nine stars and zero dependents means essentially no production user base to validate edge cases, no ecosystem of third-party strategies, and a single-maintainer bus factor. The listed last release date (2026-09-17) is in the future; this is either a data-quality anomaly in the registry feeding or a misconfigured release tag, and it should be verified before we treat the project as actively maintained. Secondary risks: (a) no visible PHP version constraint in the provided material, so we cannot confirm compatibility with the PHP 8.1/8.2/8.3 matrix that current Laravel versions support; (b) TokenLimitPostProcessor is named around token budgets without evidence it uses a real tokenizer (tiktoken-equivalent) — approximation errors here directly translate into truncated prompts or wasted context window; (c) RegexChunkingStrategy accepts user-supplied patterns, which introduces ReDoS exposure if patterns ever become configurable by non-developers; (d) the mutable TextChunker input (setText/setFile) and reset(): void on strategies imply instance state that must be reset between requests under Octane.
Key questions to resolve before commitment.
composer.json and CI?TokenLimitPostProcessor tokenizer-aware, and which tokenizer does it assume? If approximate, what is the observed drift vs. our target model?HtmlChunkingStrategy's XPath mode handle malformed or deeply nested markup?Stack fit. Native fit for a PHP 8+ / Laravel backend. It slots into the document-ingestion side of a RAG pipeline: read → chunk → post-process → embed → store. It complements rather than overlaps with vector stores, embedding clients, and LLM SDKs. It does not solve tokenization-to-model mapping, embedding batching, or retrieval ranking — those remain ours to own. For non-Laravel PHP services (e.g., a standalone worker), the library works unmodified, which is a genuine advantage if our architecture splits ingestion from the Laravel monolith later.
Migration path. Assume greenfield: adopt directly inside a dedicated ChunkingService wrapper rather than calling TextChunker from jobs or controllers. That wrapper is our seam — it lets us swap strategies, add caching, emit metrics, and later replace the library without touching call sites. If we already have a homegrown splitter or a competing package in place, migrate per-strategy rather than big-bang: start with SentenceChunkingStrategy + OverlappingChunkPostProcessor for prose, MarkdownChunkingStrategy for docs, RecursiveChunkingStrategy as the general fallback, and compare retrieval quality on a fixed evaluation set before cutting over.
Compatibility. Two axes to verify: framework compatibility (Laravel 10/11/12 require PHP 8.1/8.2/8.3 respectively) and PHP compatibility declared in the package. A short POC on our actual runtime — not the maintainer's — is mandatory. The library has no known conflicting dependencies, so composer require should be low-friction; confirm autoload and PSR-4 conformance during the spike.
Sequencing.
ChunkingService + a Laravel service provider, publish config for default strategy and token budget, add feature flag.MetadataEnricherPostProcessor for observability (index, total, source, word/char counts), and DeduplicationPostProcessor for repeated boilerplate.TokenLimitPostProcessor against real tokenizer counts for the target model.TextNormalizationPostProcessor and custom strategies.Maintenance. Low ongoing maintenance from us if we constrain usage to the documented surface (strategies + post-processors) and avoid relying on internal behavior. High maintenance risk from upstream: single maintainer, single-digit stars, no dependents. Mitigations: pin an exact version, enable Dependabot, keep a vendored copy of the source in a private mirror, and budget ~2–4 engineer-days per quarter to absorb a fork if the project stalls. Because the codebase is small and strategy classes are self-contained, forking is cheap — that materially de-risks abandonment.
Support. Community-only. No commercial support, no SLA, no security disclosure policy visible. For a small RAG ingestion path this is acceptable; for anything customer-facing or regulated, we should treat the package as unsupported and plan for self-sufficiency.
Scaling. The streaming design scales with worker count rather than RAM, which is the right failure profile for large documents in queue workers. Two caveats: (a) chunk-level scaling depends on our ability to parallelize across documents, not within one, so throughput is bounded by queue concurrency; (b) ChunkMergerPostProcessor and any buffering post-processor will break the streaming guarantee if misordered in the pipeline — pipeline ordering is a real operational concern and should be codified in config, not code. Under Octane, reset TextChunker state between requests.
Failure modes to instrument. Unreadable or non-UTF-8 files (LocalFileReader throws RuntimeException); pathological regex patterns causing ReDoS in RegexChunkingStrategy; HTML chunkers producing empty or tag-fragmented chunks on malformed markup; TokenLimitPostProcessor undercounting tokens and silently exceeding the model budget downstream; empty-chunk propagation to embedding APIs (mitigate with ChunkFilterPostProcessor); duplicate boilerplate inflating vector store costs (mitigate with DeduplicationPostProcessor); and strategy state leaking across jobs in long-lived workers. Each of these should surface as a metric, not a silent truncation.
Ramp-up. Low. The API is fluent and self-describing, a developer can be productive in half a day, and AGENTS.md plus per-class unit tests make the extension pattern (add a strategy, add a post-processor, add a reader) explicit. The cognitive load is in tuning strategy/pipeline combinations for retrieval quality, not in learning the library. Plan for one engineer to own it as a designated maintainer; the knowledge transfer burden is small but the bus factor on our side should not be one either.
Overall recommendation. Adopt for a bounded RAG ingestion use case, behind a service wrapper and a feature flag, contingent on the PHP/Laravel compatibility check and the tokenizer question being answered. Do not adopt as a critical-path dependency without the fork-readiness mitigations above.
How can I help you explore Laravel packages today?