yethee/tiktoken
PHP port of OpenAI tiktoken for fast tokenization. Get encoders by model or encoding, encode text to token IDs, with built-in vocabulary caching (configurable cache dir). Optional experimental FFI mode using tiktoken-rs for better performance on larger inputs.
|
Package
|
Constraint
|
|---|---|
| symfony/service-contracts | ^2.5 || ^3.0 |
|
Package
|
Constraint
|
|---|---|
| doctrine/coding-standard | ^13 |
| mikey179/vfsstream | ^1.6.11 |
| phpbench/phpbench | ^1.2 |
| phpunit/phpunit | ^10.5.45 |
| psalm/plugin-phpunit | ^0.19.2 |
| vimeo/psalm | 6.15.1 |
How can I help you explore Laravel packages today?