yethee/tiktoken
PHP port of OpenAI’s tiktoken tokenizer. Get encoders by model name, encode text to token IDs, and cache vocab files for speed. Optional experimental Rust/FFI “lib mode” for faster encoding of medium/large texts.
The first stable release. There are no changes in the public API since version 0.12.0.
The lib mode is still in an experimental state. And method Encoder::encodeInChunks() is not supported.
How can I help you explore Laravel packages today?