Tokenizer X Laravel Package

Getting Started

Install via Composer and you're immediately able to count tokens without any API key or network call — the tokenizer runs locally:

composer require rajentrivedi/tokenizer-x

The package exposes a facade, so the first thing to look at is Rajentrivedi\TokenizerX\TokenizerX. The two methods you'll use 95% of the time are count() (returns an integer) and encode() (returns the raw token array — useful when you need to slice or inspect a prompt). There's nothing to configure, register, or migrate; the service provider is auto-discovered.

Your first real use case is almost always cost/length guarding before an OpenAI call:

use Rajentrivedi\TokenizerX\TokenizerX;

$tokens = TokenizerX::count($prompt);
// then compare against your model's context window before dispatching

If you're on the 1.0.2 release, the "gpt 4" changes and README rewrite are worth a skim before you build assumptions into your code — model/tokenizer naming is the one thing that tends to shift between minor versions here.

Implementation Patterns

Wrap it in a small service, don't call the facade everywhere. The facade is convenient but couples your call sites to the package. A thin class lets you swap implementations, add caching, and mock in tests:

class TokenBudget
{
    public function fits(string $text, int $window, int $reserve = 500): bool
    {
        return TokenizerX::count($text) <= ($window - $reserve);
    }
}

Truncate by tokens, not characters. The common pattern is to encode once, slice the array, and decode — this is where encode() earns its keep:

$tokens = TokenizerX::encode($text);
if (count($tokens) > $limit) {
    $tokens = array_slice($tokens, 0, $limit);
    $text = TokenizerX::decode($tokens); // if the version you're on exposes decode
}

If decode() isn't available on your version, fall back to character-level trimming with a safety margin — but leave yourself headroom, since you can't losslessly cut mid-token.

Chunking for embeddings. Walk the document, accumulating sentences until count() approaches your chunk target, then flush. Counting each candidate chunk individually is O(n²)-ish on large inputs; instead accumulate an array and call count(implode(' ', $chunk)) once per flush, or budget by paragraph and verify.

Validation rule. For user-submitted prompts, a custom rule reads naturally:

class MaxTokens implements ValidationRule
{
    public function __construct(private int $max) {}

    public function validate(string $attribute, mixed $value, Closure $fail): void
    {
        if (TokenizerX::count((string) $value) > $this->max) {
            $fail("The {$attribute} exceeds {$this->max} tokens.");
        }
    }
}

Cache aggressively. Token counting is pure and deterministic — hash the text with xxh128 or md5 and memoize in the cache. If you're counting the same system prompt on every request (a very common pattern), this turns a per-request cost into a per-deploy cost.

Pair with cost accounting. Store a $perToken map keyed by model name and multiply — you get pre-flight cost estimates for free, which is the single most requested feature in any app that wraps OpenAI.

Gotchas and Tips

Tokenizers are model-specific, and that's the #1 source of wrong numbers. cl100k_base covers GPT-3.5-turbo and GPT-4-era models; older davinci/text-davinci-003 use r50k_base. If you're counting with the wrong vocabulary your numbers will be off by a few percent — enough to blow a context window on a long prompt. Check which encoding the version you installed pins, and if you still serve legacy models, verify you can select the encoding rather than relying on a single hardcoded default.

Chat completions cost more tokens than the text. Every message adds role/name framing tokens, plus a small priming overhead for the reply. The library counts strings, not conversations. Budget roughly 3 tokens per message plus ~3 for the reply priming, and always leave a reserve — do not compare count($text) directly against the model's context size.

Don't count inside loops over large corpora. Counting a 50-page document takes measurable time; doing it per-request for a static knowledge base is wasted CPU. Precompute at ingest, store the count in a column, and read it back.

Truncation vs. rejection are different UX decisions. Silently trimming a user's prompt is a support ticket waiting to happen. Prefer rejecting with an accurate message during validation, and reserve truncation for server-side content you control (RAG context, system prompts).

Watch the token boundaries when chunking. Slicing a token array and re-decoding can produce mangled UTF-8 if you cut in the middle of a multi-byte character's token sequence. If you're not using decode(), trim at word boundaries and then re-verify the count — cheap, and it avoids mojibake in your embedding store.

Test against real counts, not fixtures. Because tokenizers drift between model generations, snapshot a handful of representative strings and their expected counts in a test. When you bump the package, that test tells you immediately whether the underlying encoding changed.

PHP version and ext-mbstring. Multibyte handling is the whole job here; a container missing mbstring will fail in confusing ways (or silently miscount). Confirm the extension in your Dockerfile and in CI.

Version pinning. This is a young package at 1.0.x. Pin to the minor (^1.0) and read the changelog on each bump — the 1.0.2 notes are terse (a PR titled "gpt 4" plus a Dependabot bump and a README rewrite), so diff the source rather than trusting the release title if token counts matter to your billing estimates.