Tiktoken For Laravel Laravel Package

Getting Started

composer require mis3085/tiktoken-for-laravel
php artisan vendor:publish --tag=tiktoken-config   # check README for exact tag
php artisan list | grep tiktoken                   # confirm the pre-download command name

The package is a thin Laravel service-provider wrapper around tiktoken-php, so the first place to look is config/tiktoken.php (cache location, default encoding) and then the package's ServiceProvider to see exactly what is bound and under which alias. Everything else is the upstream TikToken\Tiktoken API: getEncoding($name) / getEncodingForModel($model) return an Encoding, and encode() / decode() do the real work.

Because the BPE rank files are downloaded from OpenAI's blob storage on first use, run the package's download/pre-cache Artisan command (or a one-off Tiktoken::getEncoding('cl100k_base')) during image build or deploy — not on the first user request.

The canonical first use case is a guard on user-supplied prompts, e.g. a FormRequest rule or a controller check before you call the OpenAI API:

use Mis3085\Tiktoken\Facades\Tiktoken; // verify alias from the provider

$encoding = Tiktoken::getEncodingForModel('gpt-4o');
$count = count($encoding->encode($prompt));

abort_if($count > 8_000, 422, 'Prompt too long.');

Implementation Patterns

Wrap token math in one service, not in controllers. Inject TikToken\Tiktoken (or the facade) into a small TokenCounter class so encoding selection, fallbacks, and caching live in one place. The Laravel binding is almost certainly a singleton — keep it that way; encoding objects are expensive to build.

Model → encoding mapping belongs in config. cl100k_base covers GPT-4 / 3.5-turbo / text-embedding-ada-002; o200k_base covers GPT-4o and the 4.1 family. Prefer getEncodingForModel($model) so new models work after a composer update, and keep a fallback map in config/tiktoken.php for anything it throws on:

try {
    $enc = $this->tiktoken->getEncodingForModel($model);
} catch (\Throwable) {
    $enc = $this->tiktoken->getEncoding(config("tiktoken.fallback.$model", 'o200k_base'));
}

Truncate on tokens, not characters. Encode → array_slice → decode is the standard way to fit a context window. Leave headroom: chat completions add per-message overhead (~3–4 tokens each) plus reply priming, and tool/function schemas are billed too. A max_tokens - 512 budget for the prompt is a sane default.

Chunking for embeddings. Split long documents by encoding, slice into windows of N tokens with a ~10–15% overlap, decode each slice back to text. Store the token count alongside the chunk so you can re-chunk later without re-tokenizing.

Cache counts for repeated input. Identical system prompts, templates, and tool schemas are tokenized on every request. Cache::remember('tok:'.hash('xxh3', $text), now()->addDay(), fn () => count($enc->encode($text))) pays for itself immediately.

Estimate cost next to the count. $tokens / 1_000_000 * $pricePerMillion from a config table gives product/ops a number they can act on without touching the code path.

Pre-warm in deploy, not runtime. In Dockerfile or a deploy script, call the download command and point the cache at a persisted path (storage/app/tiktoken or the upstream TIKTOKEN_CACHE_DIR). This is the difference between a 2-second cold start and a request that blocks on a network fetch.

Gotchas and Tips

• First call hits the network. In CI, in read-only containers, or behind a proxy this throws — and it looks like a tokenizer bug. Pre-download the files and bake them into the image, or mount a writable cache volume.
• Cache dir permissions. The default cache path is often a system temp dir. If your PHP-FPM user can't write there, every request re-downloads. Set an explicit path under storage/.
• getEncodingForModel() throws on unknown models. New model names ship faster than the library updates. Always try/catch and fall back rather than letting a 500 escape.
• Token count ≠ strlen() ≠ mb_strlen(). Do not approximate; the whole point of the package is exactness. It's also the only way your cost estimates match the invoice.
• Decoding a slice is not always clean. Cutting mid-merge or mid-multibyte can yield replacement characters or a dropped word. Validate the decoded string (mb_check_encoding) and trim trailing whitespace/partial tokens.
• Invalid UTF-8 throws. User input from legacy clients or DB columns can trip this; sanitize with mb_convert_encoding($s, 'UTF-8', 'UTF-8') before encoding.
• Memory in long-lived workers. cl100k_base in PHP is tens of MB of arrays. Under Octane or queue workers this is fine once, but don't re-resolve encodings per job item, and watch memory_limit if you tokenize very large documents.
• Don't mix encodings within one conversation. If you truncate with cl100k_base but bill against a o200k_base model, your budget checks are wrong. Derive the encoding once from the model and pass it down.
• Tests should not touch the network. Seed the cache in a test bootstrap, or stub your TokenCounter behind an interface — asserting exact token IDs in feature tests is brittle anyway; assert on counts and length limits.
• Version drift between the wrapper and tiktoken-php. The wrapper is thin, so the upstream release notes are where new encodings land. Pin both and upgrade deliberately when you add a model to your routing table.