Tiktoken Laravel Package

Getting Started

Install and bind it once. The package has no Laravel service provider of its own, so you wire it up manually. Do that in one place and inject the provider — don't new EncoderProvider() at call sites, because each instance re-resolves vocabularies.

composer require yethee/tiktoken

// config/tiktoken.php
return [
    'cache_dir' => env('TIKTOKEN_CACHE_DIR', storage_path('app/tiktoken')),
    'use_lib'   => (bool) env('TIKTOKEN_LIB_PATH', false),
    'default'   => env('TIKTOKEN_ENCODING', 'o200k_base'),
];

// app/Providers/AppServiceProvider.php  (register())
$this->app->singleton(EncoderProvider::class, function () {
    $provider = new EncoderProvider(config('tiktoken.use_lib'));
    $provider->setVocabCache(config('tiktoken.cache_dir'));

    return $provider;
});

Then wrap it in a thin domain service so the rest of the app never touches the encoder API:

namespace App\Services;

use Yethee\Tiktoken\EncoderProvider;

class TokenCounter
{
    public function __construct(private EncoderProvider $provider) {}

    public function count(string $text, string $model = 'gpt-4o'): int
    {
        return count($this->provider->getForModel($model)->encode($text));
    }
}

First use case — pre-flight validation on an OpenAI call. This is the 80% case: you already have a chat/completion controller, and you want to stop sending prompts that will 400 or blow the budget.

$tokens = app(TokenCounter::class)->count($prompt);

abort_if($tokens > $limit, 422, "Prompt is {$tokens} tokens; limit is {$limit}.");

Where to look first: Yethee\Tiktoken\EncoderProvider (the only class you should normally reference), the Yethee\Tiktoken\Encoder interface (it's an interface since 0.9.0, with NativeEncoder and LibEncoder implementations), and the changelog for supported model names.

Warm the vocabulary cache at deploy time. The first call downloads vocabulary files; on a cold container that's a multi-second network round trip inside a user request. Add an artisan command that calls getForModel() for each model you use and run it in your deploy pipeline:

foreach (config('ai.models') as $model) {
    $this->provider->getForModel($model); // populates the cache dir
    $this->info("cached: {$model}");
}

Implementation Patterns

Chat messages need overhead you must add yourself. The package tokenizes a string; it does not implement OpenAI's chat framing. A messages array costs roughly 3 tokens per message plus ~3 for the reply priming, on top of encode($content). Build the accounting in your TokenCounter so no controller re-derives it:

public function countMessages(array $messages, string $model): int
{
    $encoder = $this->provider->getForModel($model);
    $total = 3; // priming

    foreach ($messages as $m) {
        $total += 3 + count($encoder->encode($m['content']));
        if (isset($m['role'])) { /* role is covered by the 3 */ }
    }

    return $total;
}

A validation rule beats scattered count() calls. Make the budget declarative and reusable across FormRequests:

class MaxTokens implements ValidationRule
{
    public function __construct(private string $model, private int $limit) {}

    public function validate(string $attribute, mixed $value, Closure $fail): void
    {
        if (app(TokenCounter::class)->count((string) $value, $this->model) > $this->limit) {
            $fail("The :attribute exceeds the {$this->limit} token limit.");
        }
    }
}

Truncate by dropping whole messages, not by cutting strings. Truncating text mid-token produces mangled output and can split a multi-byte character. Walk the conversation oldest-first and drop messages until the running count fits:

while ($this->countMessages($messages, $model) > $budget && count($messages) > 1) {
    array_shift($messages); // drop oldest, keep the system prompt
}

RAG chunking: accumulate, don't slice. For embedding pipelines you want ~1k-token chunks on natural boundaries. Split on paragraphs, then accumulate until the token budget is hit — that keeps you from splitting sentences:

$budget = 1000; $chunks = []; $buffer = ''; $encoder = $provider->getForModel('text-embedding-3-small');

foreach (preg_split('/\n{2,}/', $document) as $para) {
    $candidate = ltrim($buffer . "\n\n" . $para);

    if (count($encoder->encode($candidate)) > $budget && $buffer !== '') {
        $chunks[] = $buffer;
        $candidate = $para;
    }

    $buffer = $candidate;
}

if ($buffer !== '') $chunks[] = $buffer;

encodeInChunks() exists on the Encoder interface but note the LibEncoder implementation is still unimplemented (explicit TODO in the README) — stick to the accumulate pattern if you're on lib mode.

Memoize on the hot path. System prompts and instructions are identical on every request. Cache the count by content hash:

return Cache::rememberForever(
    'tok:'.$model.':'.hash('xxh3', $text),
    fn () => count($this->encoder($model)->encode($text))
);

Cost estimation belongs next to the counter. Keep a price table in config and multiply — this is what makes the package pay for itself and what product actually asks for:

'pricing' => ['gpt-4o' => ['in' => 2.50 / 1_000_000, 'out' => 10.00 / 1_000_000]],

$cost = $tokens * config("ai.pricing.{$model}.in");

Log token usage for every completion. Log::info('tokens', ['model' => $model, 'in' => $in, 'out' => $out, 'user_id' => $id]) gives you the audit trail and the analytics table for free, and it's how you find the one feature burning 60% of the budget.

Queue the expensive paths. Document chunking and embedding preparation should be jobs, not request-time work. The encoder is a singleton in the worker, so it resolves once and stays warm across many jobs — that's where the package performs best.

Gotchas and Tips

The vocabulary cache is the #1 production risk. Default location is sys_get_temp_dir(). On Laravel Vapor, Lambda, or any multi-container setup, /tmp is ephemeral or per-instance, so every cold start re-downloads vocabularies — and that download is a live network call. Set cache_dir to a persistent, writable path (storage_path('app/tiktoken')) or bake the files into your image, and warm it during deploy. Also: the vocab cache is downloaded at runtime, so an environment with no outbound internet requires pre-seeding.

Pin ^1.1.1 or higher. 1.1.1 fixed a race condition in vocabulary cache updates — under concurrent first requests, workers racing to write the same vocab file were corrupting it. Anything below that is a live risk in any environment with more than one PHP process.

You cannot disable caching. As of 0.9.0, setVocabCache() takes a non-nullable string; passing null is a BC break that now errors. If you need "no disk cache," point it at a tmpfs you control rather than trying to turn it off.

Type-hint the Encoder interface, not NativeEncoder. Encoder became an interface in 0.9.0. If you type-hint concretely you'll have to rewrite call sites the day you flip on lib mode.

Lib mode is experimental — benchmark before you adopt. new EncoderProvider(true) forces the FFI LibEncoder, which requires a compiled libtiktoken_php.{so,dylib,dll} and a Rust 1.85+ toolchain to build. Data marshalling overhead makes it slower for short strings; it only wins on medium/large texts. It doesn't implement encodeInChunks(). Run composer bench against your actual prompt sizes before assuming it's an upgrade, and don't attempt it on shared hosting where you can't control the binary.

getForModel() will throw for models the package doesn't know. Model names are hardcoded in the library, so a brand-new OpenAI release (GPT-5.2 support landed only in 1.1.0) will fail before you've upgraded. Defend with a fallback to an encoding preset and keep the mapping in config:

try {
    $encoder = $provider->getForModel($model);
} catch (\Throwable) {
    $encoder = $provider->get(config('tiktoken.default')); // o200k_base
}

Special tokens are not supported. <|endofprompt|> and friends get tokenized as literal text. If you rely on custom tokens in fine-tuned models, this package is the wrong tool.

Token counts are an approximation of your bill, not the bill. Vision inputs, tool/function schemas, and reasoning tokens are all billed differently and none of them are counted here. Always leave a safety margin (5–10%) rather than sizing prompts to the exact limit.

Reuse the singleton. In Octane and long-lived queue workers a single EncoderProvider keeps vocabularies in memory, which is what you want — but budget memory_limit accordingly, since those vocab tables are large PHP arrays. Never construct a provider per tokenization call; it re-parses the vocabulary.

Extension point: if you need vocabularies from an internal mirror or an offline bundle, VocabLoader::loadFile() (added in 0.9.0) lets you load a local file instead of the remote URI — that's your hook for air-gapped deployments.