Tiktoken Laravel Package

Getting Started

Minimal Setup

1. Install the package:

   composer require yethee/tiktoken
   

2. Basic tokenization:

   use Yethee\Tiktoken\EncoderProvider;
   
   $provider = new EncoderProvider();
   $encoder = $provider->getForModel('gpt-3.5-turbo-0301');
   $tokens = $encoder->encode('Hello world!');
   // Returns: [9906, 1917, 0]
   

First Use Case: Token Counting for API Cost Estimation

$provider = new EncoderProvider();
$encoder = $provider->getForModel('gpt-4');
$tokens = $encoder->encode("Your prompt here");
$tokenCount = count($tokens);
$cost = $tokenCount * 0.00003; // Example: $0.03 per 1k tokens

First Use Case: Input Validation Middleware

use Yethee\Tiktoken\EncoderProvider;
use Closure;

class TokenLimitMiddleware
{
    public function __construct(private EncoderProvider $provider) {}

    public function handle($request, Closure $next)
    {
        $encoder = $this->provider->getForModel('gpt-4');
        $tokens = $encoder->encode($request->input('prompt'));

        if (count($tokens) > 8000) { // GPT-4 context limit
            abort(422, 'Prompt exceeds token limit');
        }

        return $next($request);
    }
}

Implementation Patterns

Core Workflow: Model-Agnostic Tokenization

// Initialize once (e.g., in a service container)
$provider = new EncoderProvider();

// Reuse encoder for the same model
$gpt4Encoder = $provider->getForModel('gpt-4');
$gpt3Encoder = $provider->getForModel('gpt-3.5-turbo');

// Encode any text
$tokens = $gpt4Encoder->encode("Your dynamic content");

// Decode back to text (if needed)
$text = $gpt4Encoder->decode($tokens);

Integration with Laravel Services

// In a service class
class PromptService
{
    public function __construct(
        private EncoderProvider $provider,
        private array $modelConfig
    ) {}

    public function validatePrompt(string $prompt, string $model): void
    {
        $encoder = $this->provider->getForModel($model);
        $tokens = $encoder->encode($prompt);

        $maxTokens = $this->modelConfig[$model]['max_tokens'];
        if (count($tokens) > $maxTokens) {
            throw new \RuntimeException("Prompt exceeds {$maxTokens} token limit");
        }
    }
}

Chunking Large Texts for Embeddings

use Yethee\Tiktoken\Encoder;

class DocumentProcessor
{
    public function __construct(private EncoderProvider $provider) {}

    public function chunkText(string $text, int $maxTokens = 1000): array
    {
        $encoder = $this->provider->get('p50k_base');
        $tokens = $encoder->encode($text);
        $chunks = [];

        foreach (array_chunk($tokens, $maxTokens) as $chunk) {
            $chunks[] = $encoder->decode($chunk);
        }

        return $chunks;
    }
}

Caching Strategies

// Configure cache directory (e.g., in config/services.php)
$provider = new EncoderProvider();
$provider->setVocabCache(storage_path('app/tiktoken-cache'));

// Or via environment variable
// TIKTOKEN_CACHE_DIR=/path/to/cache

// Cache is automatically managed; no manual invalidation needed
// (except when vocab files are updated externally)

Experimental: High-Performance Mode

// Initialize lib mode (requires Rust/FFI setup)
use Yethee\Tiktoken\Encoder\LibEncoder;

LibEncoder::init('/path/to/libtiktoken_php.so');

// Force lib mode for all encoders
$provider = new EncoderProvider(true);

// Or use selectively
$encoder = $provider->getForModel('gpt-4', true);

Gotchas and Tips

Common Pitfalls

1. Cache Directory Permissions:

   • Ensure the cache directory is writable by PHP (chmod -R 755 storage/app/tiktoken-cache).
   • Defaults to sys_get_temp_dir(), which may cause issues in shared hosting.

2. Model Name Mismatches:

   • Always use exact model names (e.g., 'gpt-3.5-turbo-0301' not 'gpt-3.5-turbo').
   • Check supported models for updates.

3. Lib Mode Overhead:

   • Avoid for small texts (<100 tokens). The FFI overhead negates performance gains.
   • Test thoroughly—experimental features may have edge cases.

4. Token Counting Quirks:

   • BOM (Byte Order Mark) can add extra tokens. Strip it first:

     $text = mb_convert_encoding($text, 'UTF-8', 'UTF-8');
     

   • Special characters may not tokenize as expected (e.g., emojis, rare scripts).

5. Race Conditions:

   • Vocabulary cache updates are thread-safe, but avoid concurrent writes if manually managing cache.

Debugging Tips

1. Verify Tokenization:

   $encoder = $provider->getForModel('gpt-4');
   $tokens = $encoder->encode("Test");
   $decoded = $encoder->decode($tokens);
   // Compare $decoded to original text for accuracy
   

2. Check Cache Issues:

   • Clear cache manually if tokens change unexpectedly:

     rm -rf storage/app/tiktoken-cache/*
     

   • Or via API:

     $provider->clearVocabCache();
     

3. Lib Mode Errors:

   • Common causes:
     • Missing library file (LibEncoder::init() not called).
     • Incorrect library path (check LD_LIBRARY_PATH).
     • Platform mismatch (e.g., Linux .so on macOS).
   • Debug:

     try {
         $encoder = $provider->getForModel('gpt-4', true);
     } catch (\Yethee\Tiktoken\Exception\LibError $e) {
         Log::error('Lib mode failed: ' . $e->getMessage());
     }
     

4. Performance Bottlenecks:

   • Profile with large texts (>10k tokens) to decide if LibEncoder is worth the setup.
   • Avoid repeated encoder creation—reuse instances (e.g., via Laravel’s service container).

Extension Points

1. Custom Vocabularies:

   • Extend Yethee\Tiktoken\Vocab\VocabLoader to load vocabularies from custom sources (e.g., S3, database).
   • Example:

     $loader = new CustomVocabLoader();
     $vocab = $loader->load('custom://vocab.json');
     $encoder = new NativeEncoder($vocab);
     

2. Token Filtering:

   • Override Yethee\Tiktoken\Encoder\NativeEncoder to filter tokens (e.g., remove stop tokens):

     class FilteredEncoder implements Encoder {
         public function encode(string $text): array {
             $tokens = parent::encode($text);
             return array_filter($tokens, fn($token) => $token !== 1234); // Example filter
         }
     }
     

3. Chunking Logic:

   • Implement encodeInChunks() for streaming or large texts (currently experimental):

     $tokens = $encoder->encodeInChunks("Very long text", 500); // 500-token chunks
     

4. Middleware Integration:

   • Create a Laravel middleware to validate token counts globally:

     public function handle($request, Closure $next) {
         $tokens = $this->provider->getForModel('gpt-4')->encode($request->prompt);
         if (count($tokens) > config('ai.max_tokens')) {
             abort(422, 'Token limit exceeded');
         }
         return $next($request);
     }
     

Configuration Quirks

1. Environment Variables:

   • TIKTOKEN_CACHE_DIR: Override cache location (e.g., /var/cache/tiktoken).
   • TIKTOKEN_LIB_PATH: Path to FFI library (e.g., /usr/local/lib/libtiktoken_php.so).
   • LD_LIBRARY_PATH: Fallback for lib mode (Linux/macOS).

2. Service Container Binding:

   • Bind EncoderProvider in AppServiceProvider for dependency injection:

     $this->app->singleton(EncoderProvider::class, function () {
         $provider = new EncoderProvider();