- How do I install tiktoken-for-laravel in a Laravel app?
- Install it with composer require mis3085/tiktoken-for-laravel. Laravel package auto-discovery registers the service provider and the Tiktoken facade, so no manual wiring is needed on supported Laravel versions. Publish the config file with php artisan vendor:publish to adjust encoding and cache settings.
- Which encodings and model identifiers does tiktoken-for-laravel support, and what happens with an unknown model?
- The package is a thin wrapper over yethee/tiktoken, so the available encodings are the BPE vocabularies that library ships, such as cl100k_base and o200k_base. Model-to-encoding resolution, and whether an unrecognized model throws or silently falls back to a default, is defined by that dependency, so test your exact model strings rather than assuming. Because newer models use o200k_base while older ones use cl100k_base, a stale mapping can quietly return wrong counts, so verify current coverage before relying on it for billing.
- Can I count tokens locally without calling the OpenAI API?
- Yes. Counting, encoding, and decoding happen entirely in PHP against local BPE rank files, so no OpenAI API key or network call to OpenAI is involved at query time. The only network dependency is fetching the vocabulary files on first use, which you should pre-cache for production deployments.
- Does the package expose encode and decode for truncation and chunking, or only token counts?
- The facade wraps tiktoken-php, which exposes encode and decode in addition to plain counts, so you get token arrays you can slice for token-accurate truncation and RAG chunk sizing. That is the main advantage over strlen()/4 heuristics: you cut at real token boundaries instead of guessing. Wrap these calls in your own TokenCounter interface so the facade does not leak into domain code.
- Will the token count match what OpenAI actually bills for chat completions?
- No, a raw token count of the message text will undercount billed usage. Chat completions add per-message framing overhead plus tokens for tool and function schemas, which a plain string count does not capture. Validate the package's output against the usage object returned by the API before you build quotas or cost attribution on top of it.
- Can I use this package to count tokens for Claude, Gemini, or Llama models?
- Treat it as OpenAI-only. Tiktoken implements OpenAI's BPE encodings, so counts for Claude, Gemini, or Llama are approximations, and tiktoken typically undercounts Claude in particular. If your roadmap is multi-provider, define a provider-specific counter interface and plug tiktoken in as the OpenAI implementation.
- How do I avoid downloading tiktoken vocabulary files at runtime in Docker or air-gapped deployments?
- Tiktoken needs BPE rank files and the underlying library may download them on first use, so confirm where they are written and whether they persist across deploys. For containers, autoscaled workers, or air-gapped environments, pre-bake the files into your image or a mounted cache path and point the config at it. Also check that concurrent workers do not race on the same download path, since an implicit runtime fetch is a deployment blocker rather than a detail.
- Does tiktoken-for-laravel work with Laravel Octane and queue workers?
- It is a stateless utility with no persistence, queueing, or business logic, so it is safe under Octane and inside queue workers from an application-state perspective. The real concern is the vocabulary file lifecycle: make sure every worker process has the rank files available locally instead of each one attempting a first-use download. Test under your actual worker concurrency before rolling it out.
- How do I estimate OpenAI API costs per request or per tenant using token counts?
- Count the prompt and expected completion tokens, then multiply by the per-million-token prices for your model to get a per-request estimate. For per-tenant attribution, log the token counts alongside a tenant identifier and aggregate them outside the request path. Keep the counter behind your own interface so you can swap implementations or move to a sidecar later without touching domain code.
- Is tiktoken-for-laravel actively maintained enough to use in production?
- It is a small MIT-licensed wrapper with a single maintainer and a modest star count, which is a bus-factor-1 signal you should budget for explicitly. The mitigating factor is its genuinely small surface area: vendoring or forking it is cheap, and the underlying yethee/tiktoken library does the heavy lifting. Pin the version in composer.json, check the release cadence over the last 12 months, and avoid letting the facade spread through your codebase so swapping it out stays easy.