- How do I install tiktoken-php in a Laravel project?
- Run `composer require yethee/tiktoken` in your project root. The package has zero external dependencies and works with Laravel’s autoloader. No additional configuration is needed for basic usage, though you may want to set up caching or middleware later.
- Which Laravel versions and PHP versions does this package support?
- The package supports PHP 8.1+ and is framework-agnostic, but designed for Laravel’s service container. It has no Laravel-specific dependencies, so it works in any PHP 8.1+ environment. Tested with Laravel 9.x and 10.x in CI/CD.
- Can I use this for GPT-4/5 token counting to enforce API limits?
- Yes. Use `EncoderProvider::getForModel('gpt-4-1106-preview')` to get the encoder, then call `encode()` on your prompt. Compare the token count against your budget (e.g., 8,192 for GPT-4) and reject or truncate inputs exceeding limits.
- How do I configure the vocabulary cache for production?
- Set the `TIKTOKEN_CACHE_DIR` environment variable to a persistent directory (e.g., `/var/cache/tiktoken`). Alternatively, call `$provider->setVocabCache('/custom/path')` in your service provider. Avoid the default temp dir in production to prevent cache bloat.
- Does this package support chunked encoding for large documents?
- Not yet. The `encodeInChunks()` method is unimplemented (planned for v1.0.0). For now, process documents in smaller batches manually or use the experimental LibEncoder for high-throughput scenarios, though it requires Rust/FFI setup.
- What’s the performance difference between the native encoder and LibEncoder?
- LibEncoder (Rust-backed) offers better throughput for large inputs (>10k tokens) but adds ~50ms overhead per call for small texts. Benchmark with `composer bench` before enabling it. Avoid LibEncoder in production unless you’ve validated its stability for your workload.
- How can I integrate token counts into Laravel logging?
- Log token counts using Laravel’s `Log` facade. Example: `$tokenCount = $encoder->encode($text)->count(); Log::info('Prompt tokens used', ['count' => $tokenCount, 'model' => 'gpt-4']);`. Pair with Laravel’s `Log::channel('sentry')` for observability.
- Are there alternatives if I need GPT-2 or custom tokenization?
- This package doesn’t support GPT-2 or models requiring special tokens (e.g., `<|endofprompt|>`). For those, use the official OpenAI SDK (`openai-php/client`) or a custom solution like `spatie/array-to-xml` for proprietary logic. Check the [OpenAI tokenizers repo](https://github.com/openai/tiktoken) for updates.
- How do I validate token counts in Laravel middleware?
- Create middleware like this: `public function handle($request, Closure $next) { $encoder = app(EncoderProvider::class)->getForModel('gpt-4'); $tokens = $encoder->encode($request->prompt); if ($tokens->count() > config('gpt.max_tokens')) { abort(422, 'Prompt too long'); } return $next($request); }`. Register it in `app/Http/Kernel.php`.
- What’s the maintenance status of this package?
- Active. The last release was March 2026, with frequent updates for new OpenAI models (e.g., GPT-5.x). Monitor the [GitHub repo](https://github.com/yethee/tiktoken-php) for breaking changes. The API is stable since v1.0.0, but cache dir requirements changed in v1.1.1.