Tntsearch Laravel Package

Getting Started

TNTSearch is an inverted-index search engine that lives inside your app: no Elasticsearch cluster, no Algolia bill, no extra service to run. It builds an index file from a SQL query (or the filesystem) and answers queries in ~1-3ms against it. For a Laravel app it's the pragmatic answer to "I need good site search and I don't want to operate anything" — typo tolerance, stemming, highlighting and prefix matching included.

Pick it when: you have a handful of searchable models, tens of thousands to a few million documents, a MySQL/Postgres/SQLite source DB, and you want relevance ranking without infra. Don't pick it when: you need facets/aggregations, distributed indexing, per-field boosting out of the box, or real-time multi-writer indexing at high write concurrency.

Install and choose your integration path

composer require teamtnt/tntsearch
composer require teamtnt/laravel-scout-tntsearch-driver   # if you want Scout integration

You have two ways in, and picking one on day one matters:

Most projects end up with the Scout driver for primary content and the raw API for a couple of specialized cases. That's fine — they share the same index format.

Configure

Publish the driver config and set the driver:

php artisan vendor:publish --provider="TeamTNT\Scout\TNTSearchScoutServiceProvider"

SCOUT_DRIVER=tntsearch

Then in config/scout.php, the tntsearch block is your control panel. Read the published file first — it documents the keys you'll actually touch (storage, fuzziness, fuzziness.prefix_length, fuzziness.max_expansions, fuzziness.distance, asYouType, searchBoolean, tokenizer, stemmer, stopwords). Everything lives under a storage path of your choosing:

'tntsearch' => [
    'storage'  => storage_path('indexes'),   // must be writable by php-fpm AND cli
    'fuzziness' => env('TNTSEARCH_FUZZINESS', true),
    'asYouType' => false,
    'searchBoolean' => env('TNTSEARCH_BOOLEAN', false),
],

Where to look first, in order: (1) the published config/scout.php, (2) your model's toSearchableArray() — that array is the document text, (3) the storage/indexes directory — the .index files are the entire state of your search, (4) php artisan scout:import "App\Models\Article".

First use case: searchable articles

use Laravel\Scout\Searchable;

class Article extends Model
{
    use Searchable;

    public function searchableAs(): string
    {
        return 'articles';   // -> storage/indexes/articles.index
    }

    public function toSearchableArray(): array
    {
        return [
            'id'      => $this->id,
            'title'   => $this->title,
            'excerpt' => $this->excerpt,
            'body'    => strip_tags($this->body),
        ];
    }
}

php artisan scout:import "App\Models\Article"

$articles = Article::search('romeo juliet')->paginate(15);   // ordered by relevance

That's the whole loop. Everything below is refinement.

Implementation Patterns

1. The Scout path is the default — lean on it

Model::search() returns a hydrated, relevance-ordered Eloquent collection. Use the Scout query API for the common cases and only drop to raw() when you need scores:

Article::search('laravel search')->paginate(20);
Article::search($q)->take(5)->get();                    // typeahead
Article::search($q)->where('status', 'published')->get(); // simple equality filters
$raw = Article::search($q)->raw();                      // ['ids'=>..., 'hits'=>..., 'docScores'=>...]

where() in the driver is applied around the engine (IDs filtered afterwards), so it does not improve engine performance and won't behave like a full query builder. For real filtering, use the hybrid pattern below.

2. Denormalize into toSearchableArray() — this is your relevance tuning

TNTSearch concatenates the document's fields into one blob of text. There is no per-field boost. The idiomatic Laravel workaround is field weighting by duplication:

public function toSearchableArray(): array
{
    return [
        'id'    => $this->id,
        'title' => trim(str_repeat($this->title . ' ', 3)),  // ×3 term frequency
        'body'  => $this->body,
    ];
}

Because ranking is BM25/TF-IDF over term frequency, repeating the title pushes title matches up the results. It's crude, it's effective, and it's the single highest-leverage thing you can do for search quality here. Also strip HTML, normalize case, and resolve relationships ($this->author->name, $this->tags->pluck('name')->implode(' ')) inside this method so authors and tags are searchable.

3. Raw indexer for anything that isn't one table

The indexer takes a raw SQL string, not an Eloquent builder. When your searchable document spans joins, create a view or write the SQL:

$tnt = new TNTSearch;
$tnt->loadConfig([
    'driver'   => 'mysql',
    'host'     => config('database.connections.mysql.host'),
    'database' => config('database.connections.mysql.database'),
    'username' => config('database.connections.mysql.username'),
    'password' => config('database.connections.mysql.password'),
    'storage'  => storage_path('indexes'),
    'stemmer'  => \TeamTNT\TNTSearch\Stemmer\PorterStemmer::class,
]);

$indexer = $tnt->createIndex('articles.index');
$indexer->query("
    SELECT a.id,
           CONCAT(a.title, ' ', a.title, ' ', a.excerpt, ' ', a.body,
                  ' ', GROUP_CONCAT(t.name SEPARATOR ' ')) AS content
    FROM articles a
    LEFT JOIN article_tag at ON at.article_id = a.id
    LEFT JOIN tags t ON t.id = at.tag_id
    WHERE a.status = 'published'
    GROUP BY a.id
");
$indexer->disableOutput(true);
$indexer->run();

Note CONCAT(...) giving you the weighting trick in SQL, and the WHERE letting you keep drafts out of the index entirely. First column must be the primary key.

4. Keeping the index in sync

Four mechanisms, use all of them:

• Model events — the Searchable trait hooks saved/deleted. Free for single-record CRUD.
• Model::query()->update([...]), DB::table(...)->update(), and bulk imports bypass model events. This is the #1 source of "search shows stale data" bugs. Any mass update path needs an explicit reindex or a queued job.
• Queue it. Don't do it inline. Index writes hit a SQLite file; doing that inside a web request adds latency and risks lock contention. Configure Scout's queue (SCOUT_QUEUE=true) and make sure your queue worker is running new code after deploys.
• Full rebuild on deploy for anything structural (new field added to toSearchableArray, tokenizer/stemmer change, schema change):

// In a deploy script or an artisan command
Artisan::call('scout:flush', ['model' => \App\Models\Article::class]);
Artisan::call('scout:import', ['model' => \App\Models\Article::class]);

For large tables, chunk and reindex yourself rather than relying on a single pass, and run the rebuild out-of-band.

5. Zero-downtime rebuild (swap the index file)

A rebuild from scratch means a window where the index is empty or half-built. Build to a temp name and rename() atomically:

$tmp = $tnt->createIndex('articles-tmp.index');
$tmp->query($sql);
$tmp->run();

rename(storage_path('indexes/articles-tmp.index'), storage_path('indexes/articles.index'));

Do the swap from a queued job, not from a request. rename() on the same filesystem is atomic, so searches either see the old index or the new one.

6. Search + filter hybrid (the pattern you'll use most)

Since the engine can't facet, do text-first then filter:

$raw   = Article::search($q)->take(200)->raw();
$ids   = $raw['ids'];

$articles = Article::query()
    ->whereIn('id', $ids)
    ->where('status', 'published')
    ->whereIn('category_id', $categoryIds)
    ->get()
    ->sortBy(fn ($a) => array_search($a->id, $ids))   // restore relevance order
    ->values();

Over-fetch (take(200)) because filtering will drop rows, then trim with ->take(20). If you're doing this in raw SQL instead of a collection, preserve order with ORDER BY FIELD(id, ...) on MySQL or array_search in PHP. Never trust whereIn() to come back in relevance order.

7. Typeahead: throttle, debounce, and turn on prefix matching

// config
'asYouType' => true,
'fuzziness' => false,   // prefix matching is usually enough for short input

// front-end: 250ms debounce, abort in-flight request

Expose a dedicated /api/search/suggest route returning id + title + highlighted title only — do not hydrate full models for suggestions. Rate-limit it (throttle:60,1); it's the endpoint most likely to be hammered.

8. Highlighting and snippets are built in — use them

$title   = $tnt->highlight($article->title, $query, 'mark', ['wholeWord' => true]);
$snippet = $tnt->snippet($query, strip_tags($article->body));

Render the snippet rather than a truncated excerpt — it centers on the match, which is what makes search results feel good. Escape the source text before highlighting, or you'll be injecting <mark> into user HTML.

9. Boolean search for an "advanced" UI

Not exposed through the Scout query builder in a friendly way — reach for raw TNTSearch:

$tnt->selectIndex('articles.index');
$res = $tnt->searchBoolean('laravel -vue or livewire', 50);

Space = AND, or/| = OR, -term/~term = NOT, parentheses group. This is a cheap way to offer power-user syntax without building a query parser.

10. Testing and local dev

Indexes are files, not a Laravel disk, so Storage::fake() does nothing. Give tests their own storage directory and rebuild in the test bootstrap:

// phpunit.xml or TestCase::setUp()
config(['scout.tntsearch.storage' => storage_path('indexes/testing')]);

Then index a small fixture set once per test class (or use RefreshDatabase + scout:import on seeds). Assert on IDs and ordering, not on scores — BM25 numbers shift as the corpus changes and make flaky assertions. Keep the testing index directory in .gitignore and clear it in tearDown.

Gotchas and Tips

You get IDs, not rows. search() and searchBoolean() return ['ids' => [...], 'hits' => n, 'docScores' => [...]]. Any code that treats the result as models is wrong. When you go raw, you own re-ordering and hydration — this is where most "search order is wrong" bugs come from.

search() is OR-ish, searchBoolean() is strict. A multi-word search() sums per-term BM25 contributions, so a document matching one term can outrank a document matching all of them. Users notice this immediately ("why does this result not contain my second word?"). Use searchBoolean() with space-separated terms when you want AND semantics.

Stemmer and tokenizer are baked into the index at build time. Changing them — or adding a stopword list — requires a full reindex. Searching uses whatever is stored in the index, so editing config/scout.php after the fact silently does nothing. Fuzziness, asYouType and limits are search-time and safe to change live.

The first column of your query is the primary key, and it isn't searchable by default. If users search by SKU, order number or slug and get nothing, you're hitting this. Either $indexer->includePrimaryKey() or concatenate the ID into the searchable text field.

With fuzzy_prefix_length = 2, typos near the start of a word never match. yon won't find yann. Lower the prefix length for short-word corpora (names, product codes, tags) — at the cost of more candidate expansions and slower queries. Tune max_expansions down if fuzzy search gets slow.

SQLite is single-writer. Concurrent index writes will lock and fail. Mitigations: queue all indexing, use run() (batched transactions) for bulk builds, keep WAL on (the default), and switch to MysqlEngine if you genuinely have parallel writers or multiple app servers that each need to write. RedisEngine is the option for ephemeral/fast indexes where losing the index is cheap.

The storage directory must be writable by both php-fpm and the CLI user. It is auto-created since v5.2, but with the wrong owner. This is the most common post-deploy failure, and it presents as a confusing "index not found" rather than a permission error. Check it in your deploy script.

searchableAs() is the file name. Renaming an index (e.g. articles → articles_v2) strands the old file; rename or delete it, and flush on scout:flush.

Soft deletes don't always reach the index. Scout removes on deleted, but SoftDeletes fires deleted too — so decide explicitly whether soft-deleted rows should remain searchable, and if not, verify it with a test. Restoring is the case that usually breaks.

Short tokens and stopwords are dropped. Queries of one or two characters, or common words you've added to stopwords, return nothing useful. If you need SKU/code lookup, use a custom tokenizer (ProductTokenizer) rather than fighting the default one.

Debugging checklist, in order: totalDocumentsInCollection() — is the index actually populated? File mtime in storage/indexes — was it rebuilt after your change? selectIndex() called? Does the query work with searchBoolean() (rules out ranking problems)? Inspect the raw SQLite index with the sqlite3 CLI to see what terms were actually stored — this ends most stemming/tokenizer arguments immediately.

Search is fast; hydration is the real cost. A 2ms engine query followed by a 200ms Eloquent hydrate is normal. Select only the columns you render, paginate aggressively, and don't with() heavy relations on result pages.

Config caching bites env-driven settings. If tntsearch.storage or fuzziness reads from env(), run php artisan config:cache as part of every deploy — a stale cache pointing at the old storage path is a silent zero-results bug.

There's no facet support, and there won't be. Plan your filter UI around database queries against the returned IDs, or accept a two-step "search then narrow" UX. If you find yourself wanting counts-per-category from the search engine, that's the signal you've outgrown TNTSearch.