yethee/tiktoken
PHP port of OpenAI tiktoken for fast tokenization. Get encoders by model or encoding, encode text to token IDs, with built-in vocabulary caching (configurable cache dir). Optional experimental FFI mode using tiktoken-rs for better performance on larger inputs.
|
Package
|
Description
|
Stars
|
Likes
|
Forks
|
Downloads
|
Issues
|
Score
|
Opportunity
|
License
|
Last Release
|
|
|---|---|---|---|---|---|---|---|---|---|---|---|
| benbjurstrom/markdown-object | Intelligent Markdown chunking for LLM/RAG workflows. Preserves headings, lists, tables, and semantic relationships while splitting into token-aware chunks sized for embedding model context windows. Built on League CommonMark with TikToken support. | 22 | 23 | 0 | 3K | 1 | 10.6 | 30.5 | MIT | 7 months ago | |
| droath/laravel-text-chunker | Flexible Laravel text chunking for AI/LLM apps. Split content into smaller chunks by characters, tokens, sentences, or markdown-aware rules. Fluent, strategy-based API ideal for fitting token limits, RAG pipelines, and custom domain splitting. | 3 | 3 | 1 | 77 | 0 | 0.0 | 18.0 | MIT | — |
How can I help you explore Laravel packages today?