Tiktoken Laravel Package

Change log

Added

• Added support for GPT-5.1

Added

Added support for the following models: 04-mini, gpt-oss, gpt-4.5 and gpt-5

Added

• Added support for GPT-4.1 models (close #28)

Added an alternative implementation of the encoder using tiktoken-rs library (#27).

Added

• Added Yethee\Tiktoken\Exception\IOError class
• Added Yethee\Tiktoken\Exception\LibError class
• Added Yethee\Tiktoken\Encoder\LibEncoder class
• Added Yethee\Tiktoken\Encoder\NativeEncoder class
• Added Yethee\Tiktoken\Vocab\VocabLoader::loadFile() method

Changed

• Yethee\Tiktoken\Encoder is now an interface
• Added argument $useLib to Yethee\Tiktoken\EncoderProvider constructor

  - public function __construct()
  + public function __construct(private bool $useLib = false)
  

• [BC break] You can no longer disable cache for vocabs. Changes signature of EncoderProvider::setVocabCache() method:

  - public function setVocabCache(string|null $cacheDir): void
  + public function setVocabCache(string $cacheDir): void
  

Added

• Added support for o3 models

Added

• Add support o1 models (#18, #19)
• Add support chatgpt-4o models

Added

• Added support for models:
  • babbage-002
  • davinci-002
  • text-embedding-3-small
  • text-embedding-3-large

Fixed

• Fix invalidate cache of vocab when the hash is mismatch.

Added

• Added support for encoding o200k_base (GPT-4o) (#14)
• Invalidate cache of vocab by checksum

Changed

• [BC break] Changed signature of method VocabLoader::load

  - public function load(string $uri): Vocab;
  + public function load(string $uri, string|null $checksum = null): Vocab;
  

Changed

• Improved performance of converting text into tokens (PR #10)

Removed

• [BC break] Removed method EncodeUtil::fromBytes()

Added

• Added Encoder::encodeInChunks() method, which allows encode a text in chunks.

Cache is enabled by default to avoid performance issues (see #2 and #5)