Product Decisions This Supports
- Token budgeting and spend control for AI features. This package gives you a deterministic, local token count before you call OpenAI, so you can decide to truncate, summarize, reject, or downgrade a request before paying for it. That turns "our OpenAI bill is a mystery" into a policy: max tokens per request, per user, per feature, enforced in code rather than by hope.
- Build vs. buy for tokenization. Tokenization is a deceptively fiddly piece of infrastructure (vocabulary files, encoding variants per model family, cache invalidation when OpenAI ships a new encoding). Buying it as a one-line Composer dependency avoids owning a compatibility treadmill that delivers zero differentiation to customers. Build only if we need behavior this package explicitly doesn't support.
- Quota, tiering, and monetization design. Pricing tiers ("Free: 10k tokens/month", "Pro: 500k") need a metering unit. Token counts from this package are the natural billing/limit primitive; without local counting we'd be reconciling usage from provider-side reports after the fact, which is too late to enforce anything.
- Prompt UX and guardrails. Enables concrete product behaviors: a live "you're at 85% of the limit" indicator in the composer, graceful trimming of older turns in a multi-turn chat, and rejecting pathological inputs with a friendly error instead of a provider-side 400/429 that looks like an outage to the user.
- RAG and document pipeline design. Chunk sizing for embeddings is a token problem, not a character problem. This lets us define chunking by tokens (with overlap) so chunks actually fit the embedding model's window, and validate before indexing rather than discovering truncation later.
- Observability and cost attribution. Logging token counts per feature, per tenant, per route is what makes the "which feature is eating the budget?" conversation answerable. That's a prerequisite for roadmap prioritization of AI features and for any internal chargeback model.
- Model roadmap flexibility. Support tracks current model families (GPT-4.1, gpt-4.5, gpt-5.x, o-series,
gpt-oss, embeddings), so adopting a newer model doesn't force a tokenizer migration. This de-risks the "should we switch models to cut cost?" decision — switching becomes a config change, not an infrastructure project.
When to Consider This Package
Adopt when:
- You're on Laravel/PHP and calling OpenAI directly (SDK, HTTP client, or a wrapper) for chat, completions, or embeddings, and you need token counts locally — synchronously, with no extra network round trip, inside a request or job.
- Cost control, input validation, or usage metering is a real requirement, not a nice-to-have. If you've ever shipped a prompt that occasionally blows the context window, this is the fix.
- You need token-aware UI in a request/response cycle (form validation, live counters) where calling an external service to count tokens would be absurd.
- Your volume is moderate to bursty rather than sustained high-throughput: interactive chat, document ingestion jobs, nightly batch summarization. Plain-PHP encoding is fine at this scale.
- You want minimal operational surface: MIT license, Composer install, no daemon, no sidecar, no language runtime added to the deploy image.
- You're building for multiple model families and don't want to maintain a mapping of model → encoding yourself.
Look elsewhere or plan around it when:
- You're not (only) on OpenAI. This is OpenAI encodings. If the roadmap includes Anthropic, Gemini, or open-weight models with different tokenizers, this package solves one slice and you'll want an abstraction layer in front of it regardless.
- You need special tokens or GPT-2 encodings. Explicitly unsupported. Fine for normal chat/completion flows; a blocker if you're doing fine-tuning work or anything that depends on control-token semantics.
- Throughput is the bottleneck. The FFI "lib mode" (backed by
tiktoken-rs) is the performance path, and it is experimental, requires building/shipping a native binary, and needs FFI enabled in the PHP runtime. Treat it as an optimization to reach for after measurement, not a day-one decision. encodeInChunks() is also unimplemented in that mode.
- Ops constraints are tight. The encoder downloads vocabulary files and caches them to disk. On ephemeral/containerized infrastructure that means choosing a cache directory (
TIKTOKEN_CACHE_DIR / setVocabCache()), making it writable, and deciding whether to warm it at build time — otherwise your first request after deploy pays a latency and network penalty. A race condition in cache updates was fixed in 1.1.1, which is a reminder that concurrent deploys touching a shared cache are a real failure mode to design for.
- Supply-chain risk tolerance is low. 166 stars and zero dependents means effectively no crowd-sourced validation and a single-maintainer bus factor. That doesn't disqualify it — it's a small, well-scoped port with a changelog and tests — but it argues for wrapping it behind our own interface so it's swappable.
Mitigation if we adopt: put it behind a thin internal TokenCounter contract bound as a singleton in a service provider. Costs an afternoon, removes the lock-in, gives us a place to add caching/memoization, and makes the "what if this package is abandoned?" question a non-issue in review.
How to Pitch It (Stakeholders)
To executives — one paragraph: "Every AI feature we ship is billed by the token, and today we find out what something cost after we've paid for it. This is a small, free, MIT-licensed library that lets our application count tokens locally before the request goes out. Practically, that means we can enforce per-user and per-feature limits, stop oversized requests from failing in front of customers, and give users honest feedback about how much of their allowance they've used. It's a one-line install with no new infrastructure or vendor contract, and it's the enabling piece for any usage-based pricing we want to launch. The main caveat is that it's a small open-source project, so we'll isolate it behind our own interface — a few hours of work that keeps us free to swap it later."
To engineering — the short version: "composer require yethee/tiktoken. Get an encoder by model (getForModel) or encoding (get), call encode(), get token IDs, count them. No Laravel service provider ships with it, so we bind EncoderProvider as a singleton ourselves and wrap it in a TokenCounter interface. Two things to get right up front: point the vocab cache at a writable, persisted path and warm it during deploy, and don't touch the FFI LibEncoder until we have a benchmark showing plain PHP is actually the bottleneck — it's experimental and adds a Rust build artifact to the deploy pipeline. Known gaps: no GPT-2 encodings, no special-token support, encodeInChunks() unimplemented in lib mode. None of those affect our current use cases."
To the data/ML side: "Token counts will be generated by the same encoder OpenAI uses, deterministically and locally, so chunk boundaries, cost attribution, and prompt-length validation all agree with what the model actually sees. We can pre-tokenize and chunk documents at ingestion time by token count rather than character count, which removes a class of silent truncation bugs in the embedding pipeline."
To product/design: "This unlocks the prompt-budget UX we've been sketching — live counters, soft warnings, graceful trimming of long conversations — without a spinner or an extra API call. It also means 'you've hit your limit' becomes a designed state instead of an error page."