Ai Elasticsearch Store Laravel Package

Technical Evaluation

Architecture fit

• Nature of the artifact: This is a library, not a Laravel package. It ships no service provider, no config file, no facade, and no publishable migrations. Laravel adoption is therefore a build-it-yourself integration exercise against symfony/ai-store abstractions, not a composer require and go.
• Position in the stack: It is a bridge — symfony/ai-store defines the vector-store contract (StoreInterface), and this package implements it against Elasticsearch's dense_vector field type and kNN search. Value only exists if we adopt the Symfony AI store abstraction upstream; otherwise the package has no entry point in our codebase.
• Paradigm fit: Good match for semantic search, hybrid keyword+vector retrieval, and RAG pipelines where we already index documents. kNN is approximate by default, so recall/latency trade-offs are a product decision, not just a config toggle.
• Filter support: The query abstraction with filter support (added v0.4.0) matters for multi-tenant and domain-scoped retrieval — user_id, tenant_id, locale, published_at pre-filters alongside similarity. This is the capability that usually justifies Elasticsearch over pgvector.
• Contract completeness: add, remove, query, and now clear() (v0.11.0) are implemented. remove() and clear() arriving late in the version history is a signal that lifecycle operations have been the lagging surface area — worth confirming against our deletion/GDPR requirements.

Integration feasibility

• Two viable paths:
  1. Direct Symfony AI adoption — require symfony/ai-store + this bridge, bind StoreInterface in AppServiceProvider, and use the package's own store class. Lowest code, highest coupling to Symfony AI's release train.
  2. Adapter wrapper — implement our own thin interface over the store so application code depends on an internal contract, not Symfony\Component\AI\StoreInterface. Costs roughly a day of work, buys insulation from pre-1.0 churn. Recommended for anything beyond a prototype.
• Elasticsearch prerequisite is the real cost driver: we need a cluster (self-managed, Elastic Cloud, or OpenSearch), index mappings with dense_vector and a fixed dimensionality, and an operational story for reindexing. The PHP client (elasticsearch/elasticsearch) is a separate dependency and a separate version-compatibility axis.
• PHP baseline: 8.2+. Confirm our Laravel fleet is on 8.2/8.3 before sizing effort.
• No Laravel-native configuration: base URL handling was standardized in v0.11.0, but we still own credentials, TLS, connection pooling, and timeouts in config/services.php ourselves.

Technical risk

• Pre-1.0 with a fast release cadence: v0.4 through v0.14 in the provided history, several of which are explicitly "no significant changes" (lockstep monorepo releases). Low churn per release is reassuring; the absence of a 1.0 stability guarantee is not. Pin exact versions and treat upgrades as scheduled work.
• Adoption signal is weak: 4 stars, 0 dependents. Counterweight: issues and PRs are tracked in the upstream symfony/ai monorepo, so governance and bus factor are Symfony's, not a single maintainer's. This is early-adopter territory, not abandoned territory — but we should not be the only production consumer of a code path we depend on.
• Silent-vs-loud failure semantics changed in v0.14.1: failed operations in _bulk responses now throw. This is the correct behavior and a genuine reliability improvement, but it is a runtime behavior change in a patch release. Any ingestion path we build must assume exceptions on partial bulk failure and must not treat "request returned 200" as "documents were indexed."
• Dimension immutability: dense_vector dims are fixed at index creation. Switching embedding models (768 → 1536 → 3072) means a full reindex into a new index plus a cutover. This is the single most under-estimated cost in vector search projects and should be planned for on day one, not deferred.
• Approximate recall: default kNN is approximate. For compliance, legal, or exact-match-adjacent use cases, knn: false / exact search is materially slower. Product requirements must state a recall tolerance.
• Operational familiarity gap: Elasticsearch is not a trivial dependency. Shard/replica tuning, mapping changes, snapshots, and version upgrades are ongoing work — budget for it or buy managed.

Key questions

1. Do we already run Elasticsearch/OpenSearch? If yes, this is close to free. If no, is standing up a cluster cheaper than pgvector on existing Postgres?
2. Which embedding model and dimensionality, and what is the reindex cost when we change it in 12 months?
3. Is symfony/ai-store acceptable as an application-level dependency, or do we need an internal adapter to absorb pre-1.0 churn?
4. What are the recall and p95 latency requirements? Approximate vs exact kNN is a product-level trade-off.
5. Do we have bulk ingestion (initial load, backfills, reindex)? If so, who owns retry, dead-lettering, and partial-failure observability now that _bulk failures throw?
6. Are hard-delete and tenant purge requirements satisfied by remove() and clear() as implemented, and how do we verify deletion across replicas?
7. Who on the team has run Elasticsearch in production, and what is the on-call story for cluster degradation?

Integration Approach

Stack fit

• Layer the dependency deliberately: application code → our internal retrieval interface → adapter → symfony/ai-store StoreInterface → this bridge → Elasticsearch PHP client. Collapsing the adapter layer is fine for a spike and a mistake for a product.
• Service binding: register the store as a singleton in a dedicated service provider, injecting the Elasticsearch client, index name, and vector field name from config. Keep index and field names in config, not in class constants.
• Client configuration: use Laravel's config layer for hosts, credentials, TLS, timeouts, and retry policy. Set conservative connect/read timeouts — default client timeouts are too generous for a request-path dependency.
• Index provisioning: create the index and mappings from an idempotent, versioned command or migration-like artifact, not by hand. The mapping definition (dims, similarity metric, metadata sub-fields, filterable keyword fields) belongs in version control alongside the code that assumes it.
• Bulk ingestion: run through queued jobs with explicit try/catch, per-item error extraction from the bulk response, bounded retries with backoff, and a dead-letter path. v0.14.1 makes failures visible; we still own what happens next.

Migration path

1. Spike (1–2 weeks): Docker Elasticsearch locally, one index, one use case (e.g., internal doc search), real embeddings, measure p50/p95 latency and subjective result quality against the current approach. Exit criteria: a defensible answer on recall and latency.
2. Adapter + contract: define our internal retrieval interface, implement it over the bridge, and write contract tests (add / query / filter / remove / clear) that any future backend must satisfy. This is what makes the choice reversible.
3. Shadow or dual-write: index into Elasticsearch alongside the existing store, run comparison queries offline, and only cut user-facing traffic over once quality parity is demonstrated.
4. Cutover and decommission: flip the read path behind a feature flag, keep dual-write for one release cycle, then remove the legacy path.
5. Operationalize: dashboards for query latency, empty-result rate, bulk failure count, index size, and cluster health; alerts on bulk failure rate and error-rate spikes.

Compatibility

• Laravel: any version on PHP 8.2+ is technically fine; the constraint is PHP, not the framework. Verify against our actual Laravel minor version and our container's binding/resolution behavior, since the store is not Laravel-aware.
• Elasticsearch: 7.x and 8.x are the supported range. Standardize on one major version across all environments — 7.x vs 8.x client/behavior differences are a classic source of "works in staging" failures.
• Symfony AI: the bridge tracks the symfony/ai-store release train closely (v0.11.0 and v0.12.0 were constraint-preparation releases). Upgrading one component without the other is unsupported; upgrade them together.
• v0.14.1: no API break, but the _bulk behavior change means integration tests must be updated to assert exceptions on partial failure. Any existing test that mocks a partially-failed bulk response as a success is now wrong.

Sequencing

• Confirm PHP 8.2+ and Elasticsearch 7.x/8.x standardization across environments
• Decide adopt-Symfony-AI-directly vs. internal-adapter (default: adapter)
• Stand up local and staging clusters; define index mapping and dimensionality in version control
• Run the quality/latency spike and get a go/no-go on the approach
• Implement adapter + contract tests; pin symfony/ai-elasticsearch-store to an exact version (≥ v0.14.1)
• Build ingestion jobs with bulk-failure handling, retries, and dead-lettering
• Shadow-run against the current approach; compare quality
• Cut over behind a flag; instrument; then decommission the legacy path
• Schedule quarterly dependency review for the Symfony AI release train

Operational Impact

Maintenance

• Dependency cadence: expect frequent, mostly trivial version bumps from the Symfony AI monorepo. The cost is not code churn but process — someone must own the upgrade review, and releases should be batched on a schedule rather than chased.
• Elasticsearch is the dominant maintenance burden, not this package: cluster upgrades, snapshot/restore drills, shard rebalancing, mapping changes, and reindexing. Budget it explicitly.
• Mapping and dimension drift: treat embedding model version as a first-class config, record which model produced each vector, and plan index-per-model-version so a model swap is a cutover rather than an outage.
• Pin exact versions for the bridge, symfony/ai-store, and the Elasticsearch PHP client. Floating constraints on three interdependent pre-1.0 components is how a routine composer update becomes an incident.

Support

• No first-party Laravel support surface: no vendor docs for our framework, no Laravel-tagged issues. Our internal adapter and its contract tests are the support boundary — invest there.
• Upstream support is via the Symfony monorepo (symfony/ai issues/PRs). Reasonable responsiveness is likely given Symfony's process, but there is no SLA and no commercial backing. Do not build a customer-facing commitment on this package without an internal fallback path.
• Runbook requirements: partial bulk failure triage, index reindex procedure, model-swap runbook, cluster-health degraded mode (serve keyword-only or cached results rather than erroring), and how to roll back to the previous store.
• Knowledge concentration risk: with 0 dependents there is no community corpus to search. Document the integration in-repo — the adapter, mapping, and runbooks — or we will be reverse-engineering our own decisions in six months.

Scaling

• Horizontally scalable by design via Elasticsearch shards/replicas, but the defaults are wrong for vector workloads. Size shards against vector count × dimensions × (4 bytes + overhead), and benchmark before assuming a single shard is adequate.
• kNN memory pressure: dense vectors are held in off-heap memory for fast ANN search. Vector volume growth is a capacity-planning line item, not a surprise to discover at 3× document count.
• Latency: p95 will be dominated by ANN search plus network. If we need sub-100ms with strict recall, that is a different (and more expensive) cluster shape — settle this before committing to SLAs.
• Bulk throughput: initial load and reindex jobs should be rate-limited and isolated from the query-serving path; unbounded bulk indexing will degrade user-facing latency.
• Multi-tenancy: pre-filtering by tenant in the query is cheaper and safer than index-per-tenant at our scale, but only if the filter fields are mapped as keyword from day one. Getting this wrong means a reindex.

Failure modes

• Partial bulk failure (now throwing): the highest-probability production issue. Requires per-item error parsing, bounded retry with backoff, and a dead-letter queue. Silent data loss is the failure to guard against.
• Cluster unavailable / degraded: the store has no built-in circuit breaker or fallback. We must add one, with a defined degraded behavior — cached results, keyword-only search, or a clean feature-flag-off — rather than 500s in the request path.
• Mapping conflict on write: an unexpected field or wrong vector dim produces rejected documents, which under v0.14.1 semantics surfaces as an exception. Good, but only useful if ingestion is idempotent and retry-safe.
• Recall degradation: the quietest failure. Approximate search returning mediocre results looks like success in every metric except user behavior. Track empty-result rate and click-through / follow-up-query rate as proxies, and keep an offline relevance eval set.
• Dimension mismatch after a model change: fails loudly at index time, which is preferable — but it fails at the worst time if the model change was rolled out before the index was ready. Couple the two deploys.
• Pre-1.0 dependency surprise: a minor bump changes behavior we didn't test. Mitigated by the adapter, contract tests, and exact version pins.

Ramp-up

• Two distinct skill sets required: Laravel/Symfony DI and service wiring (a day or two for a competent team), and Elasticsearch operations (weeks to real proficiency, months to comfort with failure modes). The second is the real ramp.
• Realistic effort for a first integration: 1–2 weeks for a spike, 3–5 weeks for a production-grade adapter, ingestion jobs, instrumentation, and runbooks — excluding cluster provisioning and any reindex of existing data.
• If the team has no Elasticsearch experience, either buy managed (Elastic Cloud / OpenSearch Service / a similar offering) and treat it as an appliance, or budget for a contractor/advisory engagement during the first quarter of operation. Choosing self-managed Elasticsearch with no in-house expertise is the highest-variance decision in this entire assessment.
• Ramp plan: one engineer owns the integration and becomes the Elasticsearch point of contact; adapter and contract tests are written before the second consumer appears; runbooks are validated with a deliberate failure drill (kill a node in staging, run a reindex, force a bulk failure) before go-live.