ecourty/text-chunker
ecourty/text-chunker is a PHP library for splitting long text into smaller, configurable chunks. It helps prepare content for LLM prompts, embeddings, pagination, SMS, and APIs while keeping chunks readable and within limits.
This is a framework-agnostic Composer library — there is no service provider, facade, or config file shipped. Treat it as a pure PHP dependency you wire into Laravel yourself.
composer require ecourty/text-chunker
Where to look first (in this order):
src/TextChunker.php — the entire orchestration lives in one small class. Read it once and you'll know the whole API: setFile() / setText(), withReader(), withMetadata(), withPostProcessor(), and chunk(ChunkingStrategyInterface): Generator.src/ValueObject/Chunk.php — the only thing you'll handle downstream. Immutable; withMetadata() returns a copy.src/Strategy/ — pick the strategy that matches the shape of your content, not the file extension. MarkdownChunkingStrategy for docs, HtmlChunkingStrategy for scraped pages, DialogueChunkingStrategy for transcripts, RecursiveChunkingStrategy as your safe default.src/PostProcessor/ — this is where the real value is for RAG work. Most people stop at strategies and miss that overlap, token limits, and dedupe are already built.Minimal first use case — chunk a file already in storage/app/private:
use Ecourty\TextChunker\TextChunker;
use Ecourty\TextChunker\Strategy\MarkdownChunkingStrategy;
$chunks = (new TextChunker())
->setFile(storage_path('app/private/handbook.md'))
->withMetadata(['doc_id' => 42, 'tenant' => 'acme'])
->chunk(new MarkdownChunkingStrategy());
foreach ($chunks as $chunk) {
// getters vs. public readonly props vary by version — inspect Chunk.php once
dump($chunk->getText(), $chunk->getPosition(), $chunk->getMetadata());
}
The first genuinely useful thing to build is an Artisan preview command (php artisan chunk:preview {path} {--strategy=markdown}) that dumps the first five chunks with character counts. Chunk quality is a tuning problem, and you cannot tune what you cannot eyeball.
Bind a factory, not a singleton. TextChunker is mutable and fluent — setText(), withMetadata(), withPostProcessor() all change instance state. Under Octane or in a queue worker processing many documents, a singleton will leak one document's metadata into the next. Bind a closure instead:
// AppServiceProvider::register()
$this->app->bind(TextChunker::class, fn () => new TextChunker());
Strategy selection belongs in one factory, not scattered through jobs. Route on MIME type first, extension second:
final class ChunkingStrategyFactory
{
public function forDocument(string $mime, ?string $ext = null): ChunkingStrategyInterface
{
return match (true) {
str_contains($mime, 'html') => new HtmlChunkingStrategy(/* verify ctor args in README */),
$ext === 'md' => new MarkdownChunkingStrategy(),
str_contains($mime, 'text/csv') => new LineChunkingStrategy(50),
default => new RecursiveChunkingStrategy(/* max size */),
};
}
}
Assemble the post-processor pipeline from config, in a deliberate order. The pipeline is applied in registration order, so this is a place to be explicit rather than clever:
// config/chunking.php
'pipeline' => [
TextNormalizationPostProcessor::class, // clean before measuring anything
ChunkFilterPostProcessor::class, // drop empties before they cost an embedding
DeduplicationPostProcessor::class, // md5-based; must run before overlap inflates text
ChunkMergerPostProcessor::class, // merge small fragments
OverlappingChunkPostProcessor::class, // add context tails
TokenLimitPostProcessor::class, // split whatever is still too big
MetadataEnricherPostProcessor::class, // word/char counts, index, total
];
Resolve with app($class) in a loop so any processor that later grows dependencies still works.
Queue job for ingestion. Keep the chunker inside the job, stream straight into batched embeddings, and never materialise the full set:
public function handle(EmbeddingService $embeddings): void
{
$path = Storage::disk('local')->path($this->document->path);
$chunker = app(TextChunker::class)
->setFile($path)
->withMetadata(['doc_id' => $this->document->id]);
foreach (config('chunking.pipeline') as $processor) {
$chunker->withPostProcessor(app($processor));
}
LazyCollection::make($chunker->chunk($this->strategy))
->map(fn ($c) => $c->getText())
->chunk(64)
->each(fn ($batch) => $this->document->embeddings()->createMany(
$embeddings->embed($batch->values()->all())
));
}
Wrap it behind your own interface. ChunkingService::chunk(string $path, array $meta): Generator with this package as the implementation. Costs one file, and it's the difference between a one-line swap and a refactor if the package stalls (9 stars, 0 dependents — it is one maintainer's good idea, not an ecosystem).
Custom post-processor for domain cleanup (PII redaction, redacting customer names, stripping template boilerplate) is the cheapest extension point: implement process(\Generator $chunks, string $source = ''): \Generator, yield each chunk, and do not buffer — return a new Chunk via withMetadata() rather than mutating.
chunk() returns a Generator — nothing happens until you iterate. Exceptions from a bad path, unreadable file, or malformed HTML surface inside the foreach, not at the setFile() call. Wrap the loop, not the construction.iterator_to_array() on a 200-page PDF is exactly the memory blow-up this library exists to avoid. Use LazyCollection::make() and consume once.setText() / withMetadata() persist between calls on the same instance. In Octane, queue workers, and tests using RefreshDatabase with a resolved singleton, you will see metadata from the previous document. Bind fresh or call ->setText() on a new instance.reset() exists for a reason). Reusing one strategy object across unrelated chunk() calls is only safe if the chunker resets it — verify in TextChunker.php before relying on it; when in doubt, new the strategy per document.MetadataEnricher gives you stale char counts; token-limit before overlap means the overlap pushes you back over budget. Overlap → token limit → enrich is the sane tail.OverlappingChunkPostProcessor prepends the previous tail to every chunk, so effective tokens ≈ n × (size + overlap). If you're near an API limit, budget with overlap already applied, and leave 10–15% headroom.LocalFileReader streams in bufferSize blocks; paragraph, sentence, and regex strategies receive data with $isEnd to know when to flush. A regex or paragraph break that straddles a buffer seam is the classic bug — reproduce it with a file several times larger than your buffer before you trust RegexChunkingStrategy in production.FixedSizeChunkingStrategy with real international content; if it breaks, prefer SentenceChunkingStrategy or RecursiveChunkingStrategy for those locales.TokenLimitPostProcessor is a safety net, not a tokenizer. These libraries typically approximate (~4 chars/token). Do not treat it as a hard guarantee against a provider's context_length_exceeded — validate against the real tokenizer for your model, or set the limit conservatively.HtmlChunkingStrategy leans on DOM/XPath. Malformed markup produces libxml warnings and possibly empty yields. If you're chunking scraped pages, normalize/sanitize first, and check whether tag-stripping and comment-keeping options are enabled for your use case — otherwise your embeddings carry <nav> boilerplate.Storage::path() only works on the local driver. For S3 you must either download to a temp file or implement ReaderInterface::readChunks(string $path, int $bufferSize): \Generator against Storage::readStream() and throw \RuntimeException on read failure. The custom reader is ~20 lines and unlocks the streaming benefit for remote files.position offset — the offsets are what let you later deep-link a citation back into the source document, which is the thing reviewers ask for first.composer.json (exact or ~). With a 9-star package, a minor release is an unreviewed diff; the changelog is the commit history.How can I help you explore Laravel packages today?