Tiktoken Php Laravel Package

Getting Started

composer require mehrab-wj/tiktoken-php

There is nothing else to install. This is a framework-agnostic library — no service provider, no facade, no publishable config, no artisan command, no .env keys registered for you. It will not show up in config/ or bootstrap/providers.php. That is the single biggest "where do I look" answer: everything lives in your own wrapper class.

Where to look first:

1. The README's list of supported encoding names (r50k_base, p50k_base, cl100k_base, plus a few model-specific aliases).
2. The class that loads the BPE ranks file, because that is where the network fetch and on-disk cache happen.
3. The public surface, which is a direct port of Python's tiktoken: fetch an encoding by name, then encode() (string → array of ints) and decode() (array of ints → string).

use Mehrab\Tiktoken\Tiktoken;

$enc    = Tiktoken::getEncoding('cl100k_base');
$tokens = $enc->encode('Hello, world!');

count($tokens);              // token count
$enc->decode($tokens);       // back to text

Your first real use case is the one that justifies the dependency: a pre-flight budget check before a chat completion. Before you call OpenAI, count the prompt tokens, confirm prompt_tokens + max_tokens <= context_window, and trim if not. That single guard prevents the two failure modes that actually cost money — 400 context_length_exceeded errors in production, and surprise bills on long conversations.

php artisan tinker
>>> count(\Mehrab\Tiktoken\Tiktoken::getEncoding('cl100k_base')->encode('one two three'))

Implementation Patterns

Wrap it in a singleton service. Constructing an encoder parses a multi-megabyte ranks file — this is not something you do per call. Bind once and reuse.

// app/Providers/AppServiceProvider.php
$this->app->singleton(TokenCounter::class);

// app/Services/TokenCounter.php
namespace App\Services;

use Mehrab\Tiktoken\Tiktoken;

class TokenCounter
{
    /** @var array<string, object> */
    private array $encoders = [];

    public function encoder(string $encoding = 'cl100k_base'): object
    {
        return $this->encoders[$encoding] ??= Tiktoken::getEncoding($encoding);
    }

    public function count(string $text, string $encoding = 'cl100k_base'): int
    {
        return count($this->encoder($encoding)->encode($text));
    }

    // Chat overhead: ~4 tokens per message + ~3 priming for the assistant reply.
    public function countMessages(array $messages, string $encoding = 'cl100k_base'): int
    {
        $tokens = 3;
        foreach ($messages as $m) {
            $tokens += 4;
            $tokens += $this->count($m['role'] ?? '', $encoding);
            $tokens += $this->count($m['content'] ?? '', $encoding);
        }
        return $tokens;
    }
}

Context-window fitting. Keep the system prompt, drop the oldest turns until the budget holds. Do not try to be clever by truncating mid-message on the first pass.

public function fit(array $messages, int $window, int $reserve, string $encoding = 'cl100k_base'): array
{
    $system = array_values(array_filter($messages, fn ($m) => ($m['role'] ?? '') === 'system'));
    $rest   = array_values(array_filter($messages, fn ($m) => ($m['role'] ?? '') !== 'system'));
    $budget = $window - $reserve;

    while ($rest && $this->countMessages([...$system, ...$rest], $encoding) > $budget) {
        array_shift($rest);
    }

    return [...$system, ...$rest];
}

Chunking for embeddings. text-embedding-ada-002 caps at 8191 tokens; text-embedding-3-* at the same. Split on token boundaries with overlap, not on str_split:

public function chunk(string $text, int $max = 8000, int $overlap = 200, string $encoding = 'cl100k_base'): array
{
    $enc = $this->encoder($encoding);
    $ids = $enc->encode($text);
    $chunks = [];

    for ($i = 0; $i < count($ids); $i += $max - $overlap) {
        $chunks[] = $enc->decode(array_slice($ids, $i, $max));
    }

    return $chunks;
}

Cost estimation and observability. Map model → encoding once, in a config array you own, and log estimated cost next to the actual usage object OpenAI returns. The delta between the two is your calibration signal — if it drifts, your encoding mapping or your message overhead is wrong.

Cache token counts, not tokenizers. Cache::remember('tok:' . md5($text), 3600, fn () => $counter->count($text)) is worth it for repeated system prompts, RAG context blocks, and document ingestion, where the same long strings get re-counted on every request or retry.

Integrate at the boundary. If you use openai-php/laravel, wrap the client in your own thin class rather than scattering count() calls through controllers. One place trims, one place logs, one place throws a domain exception when the input is hopelessly over budget.

Gotchas and Tips

The package is frozen at April 2023. That predates o200k_base, so it has no support for the encodings used by GPT-4o, GPT-4.1, o1/o3/o4, and the gpt-4o-mini family. Two consequences: (a) token counts for those models will be wrong, generally overcounting for CJK and non-Latin scripts, which is safe for budgeting but useless for billing reconciliation; (b) if getEncoding('o200k_base') throws, you must either patch the ranks loader and add the vocab file yourself, or keep a model → encoding map that routes unknown models to cl100k_base and accepts the drift. If accurate o200k_base counts are a hard requirement, evaluate a maintained alternative before committing.

First run hits the network. The BPE ranks are downloaded from OpenAI's blob storage and cached on disk. That means: a cold CI build can hang or fail behind an egress firewall; a fresh Docker container pays the download on first request; and ephemeral filesystems re-download constantly. Pre-warm the cache during your image build or deploy step, point the cache environment variable at a persistent path, and add the directory to .gitignore but not to your container wipe list.

Memory and speed. Loading cl100k_base parses roughly 100k merge ranks — expect tens of megabytes per encoder instance. Never construct one inside a loop. Under Octane or a queue worker the singleton stays warm (good), but watch resident memory since it never gets released between requests. And be realistic about throughput: this is a PHP port of a Rust tokenizer, so large documents take measurably longer than the Python reference. For bulk ingestion, run counting in queued jobs, not in the request cycle.

Truncating a token array and decoding can produce broken UTF-8. A token boundary is not a character boundary. decode(array_slice($ids, 0, $n)) may emit a replacement character or an invalid sequence at the cut. Validate the result (mb_check_encoding) or truncate by message rather than by token when the output is user-facing.

Counting content alone undercounts chat requests. The 4-tokens-per-message and 3-token priming overhead above are the well-known approximations for cl100k chat formats; they are not exact for every model, and they change with tool/function calling payloads, which add JSON schema tokens you should count too.

Token ≠ word ≠ byte. strlen() and mb_strlen() are not substitutes. The "1 token ≈ 4 characters" heuristic only holds for English prose; emoji, CJK, and base64 blobs run several tokens per character. Also, the library reports what this tokenizer produces for this text — streaming responses and server-side prompt templates can diverge from your local count, so reconcile against the usage field rather than trusting the estimate blindly.

Extension points are thin by design. The encoding name is your main lever; everything else — model mapping, caching, chunking, retry-on-overflow — you build. That is fine, but it means the wrapper class is load-bearing infrastructure, not glue code. Give it tests with known-count fixtures (a plain English sentence, a CJK string, an emoji, an empty string) so a future dependency bump can't silently shift your budgets, and pin the version in composer.json. With a single maintainer and 17 stars, treat this as vendored code you own.