Text Chunker Laravel Package

Product Decisions This Supports

• Build vs. buy for RAG ingestion: If your roadmap includes retrieval-augmented generation, semantic search, or an internal "chat with our docs" feature, the ingestion/chunking layer is a real engineering cost that rarely differentiates your product. A small, MIT-licensed, strategy-based chunker is a credible "buy" (well, adopt) that lets the team spend its budget on retrieval quality and UX instead of text-splitting plumbing.
• Feature scoping: The bundled strategies (paragraph, sentence, Markdown, HTML, dialogue, fixed-size, word count, regex, line, recursive) map directly to content types you likely already have — support articles, docs, transcripts, and marketing pages. That means you can ship a v1 covering multiple sources without a bespoke splitter per format.
• Roadmap sequencing for AI features: Because chunking strategy quality is one of the biggest levers on retrieval accuracy, having swappable strategies is a cheap way to run experiments (e.g., sentence vs. recursive-with-overlap) as an A/B or offline eval, rather than committing to one approach early.
• Cost and latency decisions: TokenLimitPostProcessor and OverlappingChunkPostProcessor directly affect embedding spend and context continuity. The package makes token budget a configurable product parameter instead of a hardcoded assumption, which supports decisions about summarization depth and per-query LLM cost.
• Operational/scale decisions: Generator-based streaming means large documents don't need to be fully resident in memory. That supports a "process customer uploads synchronously-ish in a queue worker" architecture rather than requiring a separate chunking service.
• Reusable internal platform: The post-processor pipeline (normalize, dedupe, filter, merge, enrich metadata) is the unglamorous 80% of ingestion work. Adopting it standardizes that logic across teams and products.
• Risk posture: MIT license, no dependents, last release dated 2026 — a permissive, dependency-free library is low legal and supply-chain friction.

When to Consider This Package

Adopt when:

• You are building RAG, vector search, or LLM preprocessing in PHP/Laravel and need to split files or strings into chunks. This is the core fit; outside PHP it is irrelevant.
• You need multiple chunking strategies and expect to iterate on them — the strategy interface and recursive composition are the main value.
• You process large files or user uploads where memory efficiency matters, and a streaming Generator pipeline is preferable to loading whole documents.
• You want standard post-processing (overlap, token limits, dedupe, metadata enrichment, whitespace normalization) without writing it yourself.
• You want to own the pipeline and extend it — adding a strategy, reader, or post-processor is a small, well-documented class.

Look elsewhere or hold when:

• Maturity risk is a blocker. Nine stars, zero dependents, and a single-maintainer profile signal a young project. If you cannot vendor or fork it comfortably, that's a real consideration.
• You need tokenizer-accurate limits. TokenLimitPostProcessor budget behavior should be validated against your actual model tokenizer; character- or word-based approximations can drift from true token counts for non-English or code content.
• You need semantic/embedding-aware chunking (e.g., splitting on topic shift via embeddings). This is text-structural, not semantic; you'd need to build on top.
• You need PDF/OCR/DOCX extraction. The readers here cover raw text/file streams; document parsing is out of scope and must be handled upstream.
• You're on a non-PHP stack, or you already depend on a mature framework ingestion component (e.g., LangChain-style tooling in Python). Duplicating capability adds maintenance surface.
• You need active vendor support, SLAs, or a security response commitment. None of that is on offer; treat it as a library you may have to maintain.

How to Pitch It (Stakeholders)

For executives (30 seconds): We need to split documents into small passages before our AI features can search or summarize them — this is unavoidable plumbing for any "ask our content" product. Rather than build and maintain that ourselves, we can adopt a small, free (MIT), dependency-free PHP library that already handles the common document types and the cleanup steps around them. It reduces time-to-first-AI-feature and keeps our embedding costs tunable. The caveat is maturity: it's an early-stage project, so we should vendor it, pin the version, and plan for the possibility that we maintain our fork. Low cost, low lock-in, moderate diligence required.

For engineering: It's a standalone PHP library with a clean shape: TextChunker takes a file or string, streams it through a pluggable ChunkingStrategyInterface via a ReaderInterface, and yields immutable Chunk value objects through a composable post-processor pipeline. Ten built-in strategies (paragraph, sentence, fixed-size, Markdown, HTML/XPath, dialogue, word count, regex, line, recursive) and eight post-processors (overlap, token limit, metadata enrichment, filtering, merging, normalization, dedupe, regex replace) cover most ingestion needs out of the box. Everything is interface-driven, so adding a strategy or processor is a single small class plus tests, and the Generator design means we never load a full document into memory. Integration is straightforward behind a Laravel service or a queued job; our work is mostly wiring, tokenizer validation, and defining which strategy fits which content type.

Recommendation: Adopt as an internal, pinned dependency behind our own thin ingestion abstraction (so the library is replaceable), validate token-limit behavior against our actual model tokenizer, and budget a small amount of ongoing ownership for maintenance or forking given the project's early maturity.