Embedding Laravel Package

Changed

• Widened the laravel/ai constraint from ^0.6 to >=0.6 <1.0. The package only touches Laravel\Ai\Ai, Laravel\Ai\Embeddings, and Laravel\Ai\Reranking — none of their public signatures changed through v0.11.0, and the full test suite (271 tests) passes unmodified against it. Kept below 1.0 since a real major release is expected to carry breaking changes; consumers pinned to ^0.6 are unaffected.

Changed

• missingEmbeddingCount() and the filter/generation query (SlotQueryPlanner) now check the resolved embedding text (toEmbeddingText(), after any per-model normalization), not just the raw source columns. eligibleForEmbedding()/v2.5.0 only catches a raw column that is literally null/empty — a column holding placeholder content (e.g. a description of "-----") passes that check but can still normalize down to nothing, and was still being counted as "missing" and dispatched for generation even though v2.5.1's guard would always skip it. SlotQueryPlanner gains missingIds(), chunking through candidates and applying the same resolved-text filter used for generation — this is now the single source of truth both the count and the query plan use, so they can never drift from what generation would actually do.

Fixed

• EmbeddingGenerator::generate() now checks the resolved embedding text before calling the AI provider, throwing EmptyEmbeddingTextException when it is blank instead of letting the provider reject an empty-string request. GenerateModelEmbedding swallows this exception so the job completes normally rather than retrying and landing in failed_jobs. Catches cases eligibleForEmbedding() (v2.5.0) cannot — a raw column can be non-blank yet still normalize to an empty string (e.g. toEmbeddingText() stripping placeholder punctuation like "-----").

Added

• scopeEligibleForEmbedding($query, $slot) on the Embeddable trait — constrains a query to records with non-blank content in at least one of the fields feeding the given slot (per embeddingSlotMap()). The AI provider rejects empty-string input, so a record whose source fields are all blank/null can never successfully embed.

Fixed

• missingEmbeddingCount() no longer counts records that could never produce embeddable text (all of the slot's source fields blank/null) as "missing". Previously every such record was counted as missing forever — generation would always fail for it, so nothing users did through the UI or CLI could ever bring the count to zero.
• SlotQueryPlanner::plan() (used by both embedding:vector:generate and BatchGenerator) now applies the same eligibleForEmbedding constraint, so these records are no longer selected for generation at all — no wasted AI API calls, no failed_jobs noise, in --force mode too.

Added

• BatchGenerator::dispatch() gains a finally parameter (?Closure $finally = null). A Batch (post-dispatch) cannot have callbacks attached after the fact — only a PendingBatch can — so this must be threaded in before the first chunk is dispatched rather than added by the caller once dispatch() returns its result.

Added

• BatchGenerator — dispatches missing-embedding generation as a trackable Bus::batch() instead of the fire-and-forget dispatch $model->embed()/embedding:vector:generate use on their own. app(BatchGenerator::class)->dispatch(Post::class, slot: null, force: false, chunk: 100) returns the Illuminate\Bus\Batch (or null when nothing is missing), so callers — a UI button, a queued orchestrator — can observe real completion via $batch->finished() / a ->finally() callback instead of guessing when the work is done. Chunks the missing-record query so memory stays bounded regardless of how many records are outstanding.
• GenerateModelEmbedding now implements Batchable, so it can be added to a Bus::batch() (required for BatchGenerator, and for any application code batching it directly).

Changed

• Extracted the missing-embedding query plan (same-connection whereDoesntHave, cross-connection pluck+reject filter) out of GenerateCommand into XLaravel\Embedding\Support\SlotQueryPlanner, shared by both the console command and BatchGenerator. No behavior change for the command.

Added

• GenerateModelEmbedding now implements ShouldBeUnique, keyed on model class + record id + slot: dispatching generation again for a record/slot that already has a job queued or processing is silently skipped instead of enqueuing a duplicate (wastes an AI API call otherwise). Different records and different slots of the same record are unaffected and continue to run in parallel. The lock is held for the job's full processing duration (success or exhausted retries), not just until it starts.
• embedding.queue.unique_for config option (EMBEDDING_GENERATE_UNIQUE_FOR env, default 3600 seconds) — safety-net expiry for the uniqueness lock in case a worker dies before releasing it. Requires a cache store that supports atomic locks (redis, database, memcached, array, file).

Added

• Static coverage counters on the Embeddable trait: Post::embeddedCount(string $slot = 'default'): int (records with a stored embedding for the slot) and Post::missingEmbeddingCount(?string $slot = null): int (records lacking the slot's embedding; without a slot, sums the missing counts across every declared slot — models with no slots defined report zero). Both handle cross-connection setups (model and embeddings table on different connections) by plucking the embedding-side ID list and verifying it against the model side instead of whereHas.

Changed

• embedding:vector:status computes its Model Coverage column via the new embeddedCount() trait method instead of a private command helper. Behaviour is identical except under a morph map: the embedding-side lookup now matches on getMorphClass() rather than the FQCN, so aliased models are counted correctly.

v2 introduces payload filtering: models can publish a set of scalar attributes ("payload") into a second embeddables table, and similarity searches can filter on them at the database level via the new filter parameter — no post-query PHP filtering, no JOIN against the application tables.

Added

• embeddables table — one row per entity holding a JSON payload column, matched to embeddings by the (embeddable_type, embeddable_id) morph pair (no FK). New Eloquent model XLaravel\Embedding\Models\Embeddable, configurable via embedding.database.embeddables_table / EMBEDDABLES_DB_TABLE.
• #[EmbedPayload] attribute (single-use, not repeatable):
  • #[EmbedPayload(['province_id', 'category_id'])] — explicit field list (strict: non-scalar values throw).
  • #[EmbedPayload('*')] / #[EmbedPayload('*', except: ['secret'])] — wildcard over the instance's attributes, excluding the primary key, $hidden, and the except list (lenient: dates serialize via serializeDate(), backed enums via ->value, incompatible values are skipped).
• toEmbeddingPayload(): array support — duck-typed (not part of HasEmbeddings), merged over the attribute-derived fields; the method wins on key conflicts.
• filter parameter (?array $filter = null) on similarTo(), similarToText() and mostSimilar(). Semantics are intentionally minimal: equality, IN (array value), AND (multiple keys). Records without a payload row never match a filtered search.
• SearchRequest DTO (XLaravel\Embedding\Contracts\SearchRequest) — carries vector, limit, threshold, ids, slot, filter.
• PayloadStore contract + DatabasePayloadStore (race-safe upsert() against the unique index; deletes on model delete).
• SyncModelPayload job — the single writer for payload rows, dispatched independently of the vector jobs on its own queue (embedding.queue.sync_payload, default embedding.sync-payload) so fast DB upserts never wait behind slow AI calls.
• $model->syncEmbeddingPayload() — synchronous payload upsert helper (no-op for models without payload definitions; works even while embedding syncing is disabled).
• Trait helpers: embeddingPayloadFields(), hasEmbeddingPayload(), resolveEmbeddingPayload(), payloadFieldsChanged(), flushEmbeddingPayloadFieldsCache().
• Embedding::payloadRecord() — convenience accessor returning the entity's Models\Embeddable row (plain method, not an Eloquent relationship — the morph pair is a composite key).
• CLI split into two namespaces mirroring the two write paths, plus umbrella commands:
  • embedding:vector:generate / embedding:vector:clear / embedding:vector:clean / embedding:vector:status — vector-side only, never touch embeddables. vector:clear keeps the --slot option; vector:clean keeps --orphans-only / --invalid-slots-only.
  • embedding:payload:sync — backfills embeddables rows without touching the AI provider or vectors; idempotent, honours --dry-run, --force refreshes existing rows, --sync upserts inline.
  • embedding:payload:clear / embedding:payload:clean / embedding:payload:status — payload-side only, never touch embeddings. payload:clean removes stale rows (class missing / row deleted / model no longer defines a payload); payload:status reports per-model payload coverage, stale rows, embedded entities missing a payload row and storage size.
• PayloadStoreMetrics contract — payload counterpart of VectorStoreMetrics (same snapshot() shape); core binds DatabasePayloadStoreMetrics (Embeddable::count() for rows, null bytes), driver packages can override for native byte figures.
• embedding:storage — cheap read-only storage snapshot of both tables (two metrics reads, no coverage / health scans); per-table Rows / Data / Index / Total plus a combined byte total (n/a unless both drivers supply bytes), --json emits {"vector": {...}, "payload": {...}}.
  • embedding:clear / embedding:clean (umbrellas) — operate on both tables for full-reset / full-cleanup. embedding:clear takes no --slot (payload is entity-level); embedding:clean takes no --*-only filters.

Changed

• Breaking: HasEmbeddings::toEmbeddingText() is now toEmbeddingText(string $slot = 'default'): string — the string|array return is gone. The model builds only the requested slot's text; multi-slot models branch on $slot (e.g. match) instead of returning every slot's text on every call. EmbeddingGenerator validates the requested slot against embeddingSlotMap() and rejects undeclared slots.
• Breaking: SimilarityDriver::search() is now search(Model $prototype, SearchRequest $request): Collection — the old 6-parameter signature is removed. Custom drivers must be updated.
• Breaking: migrations are no longer auto-loaded (loadMigrationsFrom removed). Publish them before migrating: php artisan vendor:publish --tag=embedding-migrations (or the driver package's tag when using a DB driver).
• Breaking: queue configuration split. EMBEDDING_QUEUE (default embedding) is replaced by EMBEDDING_GENERATE_QUEUE (embedding.queue.generate, default embedding.generate) for vector jobs plus EMBEDDING_SYNC_PAYLOAD_QUEUE (embedding.queue.sync_payload, default embedding.sync-payload) for payload jobs. Workers should listen to both, payload first: php artisan queue:work --queue=embedding.sync-payload,embedding.generate. SQS queue names cannot contain dots — override both envs with hyphenated names on SQS.
• Breaking: config key embedding.database.table renamed to embedding.database.embeddings_table (env EMBEDDINGS_DB_TABLE unchanged).
• Breaking: embedding:generate and embedding:status are removed — use embedding:vector:generate / embedding:vector:status (and embedding:payload:sync / embedding:payload:status for the payload side). embedding:clear / embedding:clean remain but as umbrella commands over both tables: embedding:clear no longer accepts --slot (use embedding:vector:clear --slot=...) and embedding:clean no longer accepts --orphans-only / --invalid-slots-only / --payload-only (use the namespaced clean commands).
• All six driver packages (mysql, mariadb, pgsql, oracle, sqlsrv, qdrant) require x-laravel/embedding ^2.0, adopt the SearchRequest signature, ship their own create_embeddables_table migration (same filename as core — the driver file wins) and translate filter to native JSON SQL / Qdrant payload filters.

Notes

• Choosing between where and filter: use filter for indexed / high-cardinality constraints stored in the payload; use the where closure for ad-hoc or complex Eloquent constraints against the model's own table. When both are given, both apply (no smart merging).
• Payload values are limited to scalars (int / string / bool / null) or arrays of scalars — nested structures throw (explicit field lists) or are skipped (wildcard).
• Soft deletes: with embedding.soft_delete = false (default) deleting a model removes its payload row alongside its embeddings; with true both are kept and restore leaves them untouched.

• max_length config (EMBEDDING_MAX_LENGTH) — auto-truncate input before the embedding API call.

• embedding:status shows the resolved AI provider + model in the configuration table (1.3.0); Storage section ordering (1.3.2).

• embedding:status command and VectorStoreMetrics storage-metrics contract (1.2.0).
• Cross-connection orphan / invalid-slot scans in embedding:clean (1.2.1, 1.2.2).

• Second-stage reranking via laravel/ai — Reranker service + rerankWithScores() Eloquent Collection macro (1.1.0).
• Embedding job dispatch deferred until the DB transaction commits (1.1.1).
• rerank_score set on single-item collections (1.1.2).
• embedding:clean streams IDs instead of buffering (1.1.3); embeddingSlotMap() cached per class (1.1.4).
• Soft-delete fixes: skip embed dispatch on soft-delete save, include trashed models in PhpDriver results (1.1.5, 1.1.6).
• Single-slot models reject non-default slot names (1.1.7); guard against empty AI responses in similarToText / rankByRelevance (1.1.8).

• Initial release: Embeddable trait, HasEmbeddings contract, multi-slot embeddings ($embeddable map / repeatable #[EmbedOn]), queued GenerateModelEmbedding job, VectorStore contract with JsonVectorStore default, SimilarityManager with php driver, similarTo / similarToText / mostSimilar / similarityTo / rankByRelevance, model events, soft-delete handling, embedding:generate / embedding:clear / embedding:clean commands.