yethee/tiktoken
PHP port of OpenAI’s tiktoken tokenizer. Get encoders by model name, encode text to token IDs, and cache vocab files for speed. Optional experimental Rust/FFI “lib mode” for faster encoding of medium/large texts.
|
Package
|
Description
|
Stars
|
Likes
|
Forks
|
Downloads
|
Issues
|
Score
|
Opportunity
|
License
|
Last Release
|
|
|---|---|---|---|---|---|---|---|---|---|---|---|
| benbjurstrom/markdown-object | Intelligent Markdown chunking for LLM/RAG workflows. Preserves headings, lists, tables, and semantic relationships while splitting into token-aware chunks sized for embedding model context windows. Built on League CommonMark with TikToken support. | 22 | 23 | 0 | 3K | 1 | 11.4 | 31.6 | MIT | 6 months ago | |
| droath/laravel-text-chunker | Flexible Laravel text chunking for AI/LLM apps. Split content into smaller chunks by characters, tokens, sentences, or markdown-aware rules. Fluent, strategy-based API ideal for fitting token limits, RAG pipelines, and custom domain splitting. | 3 | 2 | 1 | 13 | 0 | 0.0 | 8.7 | MIT | — |
How can I help you explore Laravel packages today?