Getting Started

Cloud Datastore is a schemaless, NoSQL document store — not a relational database — so treat this package as a secondary persistence layer alongside Eloquent, not a replacement. It shines for high-write append-only data (audit logs, activity feeds, event streams), horizontally-scaled counters, and multi-tenant key-value stores.

Install:

composer require google/cloud-datastore

The package works over REST out of the box. For production, install ext-grpc and ext-protobuf — the gRPC transport is roughly 2–3× faster per RPC and matters a lot when you're doing hundreds of lookup()s.

Where to look first, in order:

Google\Cloud\Datastore\DatastoreClient::__construct() — the full option surface (projectId, namespaceId, keyFilePath, keyFile, transport, requestTimeout, databaseId).
samples/ in the repo — short runnable scripts for every verb.
Google\Cloud\Datastore\Query — the fluent query builder; this is where 90% of your day-to-day decisions live.
Google\Cloud\Datastore\Entity — the document wrapper (ArrayAccess, not an array).

Credentials: don't hand-roll auth. Either set GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json, or let Application Default Credentials find the metadata server on GCE/Cloud Run/GKE. Only pass keyFilePath for local dev.

Emulator for local dev:

gcloud beta emulators datastore start --no-store-on-disk
export DATASTORE_EMULATOR_HOST=localhost:8081
export DATASTORE_PROJECT_ID=local-dev

The client detects DATASTORE_EMULATOR_HOST automatically and skips auth — no config changes needed.

First use case — a feature-flag / app-settings store. It exercises keys, entities, lookup, and upsert without needing a composite index, so you can ship it on day one:

// config/datastore.php
return [
    'project_id'   => env('GOOGLE_CLOUD_PROJECT'),
    'namespace_id' => env('DATASTORE_NAMESPACE'),
];

// AppServiceProvider::register()
$this->app->singleton(DatastoreClient::class, fn () => new DatastoreClient([
    'projectId'   => config('datastore.project_id'),
    'namespaceId' => config('datastore.namespace_id'),
]));

// Usage
$key  = $datastore->key('Setting', 'billing.beta');
$flag = $datastore->lookup($key);           // null when missing
$enabled = $flag['enabled'] ?? false;

$datastore->upsert($datastore->entity($key, [
    'enabled'   => true,
    'updatedAt' => new DateTimeImmutable(),
]));

Implementation Patterns

Bind the client as a singleton, always. The client holds a gRPC channel; instantiating per-request (or per-job) wastes ~50–100 ms and can exhaust file descriptors under Octane/queue workers.

$this->app->singleton(DatastoreClient::class, fn () => new DatastoreClient([
    'projectId' => config('datastore.project_id'),
    'transport' => extension_loaded('grpc') ? 'grpc' : 'rest',
]));

Model pattern — map entities to typed classes. Entity accepts a className option; the class must extend Google\Cloud\Datastore\Entity. This gives you something Eloquent-shaped without pretending Datastore is relational:

class AuditEvent extends \Google\Cloud\Datastore\Entity
{
    public function action(): string { return $this['action']; }
    public function actorId(): string { return $this['actorId']; }
}

// Write
$key = $datastore->key('AuditEvent'); // auto-generates a numeric ID
$datastore->insert($datastore->entity($key, $payload, ['className' => AuditEvent::class]));

// Read
$event = $datastore->lookup($key, ['className' => AuditEvent::class]);

Repository wrapper. Keep DatastoreClient out of controllers and jobs. A thin repository makes swapping to Firestore or BigQuery later trivial and gives you a seam for tests:

final class AuditRepository
{
    public function __construct(private DatastoreClient $ds) {}

    public function record(string $action, string $actorId, array $meta = []): void
    {
        $key = $this->ds->key('AuditEvent');
        $this->ds->insert($this->ds->entity($key, [
            'action'   => $action,
            'actorId'  => $actorId,
            'meta'     => $meta,
            'occurred' => new DateTimeImmutable(),
        ]));
    }

    public function recentForActor(string $actorId, int $limit = 50): array
    {
        $q = $this->ds->query()
            ->kind('AuditEvent')
            ->filter('actorId', '=', $actorId)
            ->order('occurred', Query::ORDER_DESCENDING)
            ->limit($limit);

        return iterator_to_array($this->ds->runQuery($q));
    }
}

Multi-tenancy via namespaces. This is Datastore's killer feature for SaaS. Each tenant gets its own namespace, and no query can accidentally cross tenants:

$ds = new DatastoreClient([
    'projectId'   => config('datastore.project_id'),
    'namespaceId' => "tenant-{$tenant->id}",
]);

Per-request instantiation is fine here because namespaces are cheap — but cache the client per namespace in a container-bound array so you don't rebuild the gRPC channel per tenant switch.

Batch operations for imports and bulk writes. Each RPC is a network round-trip; batching 500 entities into one commit is not a micro-optimization:

$entities = array_map(
    fn ($row) => $datastore->entity($datastore->key('Product', $row['sku']), $row),
    $rows
);
foreach (array_chunk($entities, 500) as $chunk) {
    $datastore->upsertBatch($chunk); // also: insertBatch, deleteBatch
}

Transactions for read-modify-write. The callback form retries automatically on contention (optimistic concurrency) — your callback must be idempotent:

$datastore->transaction(function (Transaction $tx) use ($datastore, $key) {
    $entity = $tx->lookup($key);
    $tx->upsert($datastore->entity($key, ['balance' => $entity['balance'] - 100]));
}, ['maxRetries' => 5]);

Hot counters need sharding. A single entity written from many requests becomes a contention hotspot and will throw ABORTED under load. Shard the counter across N entities and sum on read:

$shard = $datastore->key('CounterShard', random_int(0, 9));
// increment in its own transaction, then read all 10 and sum

Ancestor queries when you need strong consistency. Datastore queries are eventually consistent unless restricted to an entity group via hasAncestor(). For "list all comments on this post, right now", make comments children of the post key:

$postKey = $datastore->key('Post', $postId);
$q = $datastore->query()->kind('Comment')->hasAncestor($postKey);

The tradeoff is a hard write rate limit (~1 write/sec per entity group), so only use it where consistency is non-negotiable.

Pagination with cursors, not offsets. Datastore has no OFFSET. Use cursors:

$iterator = $datastore->runQuery($q->limit(50));
$page = iterator_to_array($iterator);
$nextCursor = $iterator->info()['endCursor'] ?? null;

// next request:
$q->start($nextCursor);

Session driver. The package ships DatastoreSessionHandler, which is a legitimate drop-in if you want sticky-session-free scaling without Redis:

// bootstrap/app.php or a service provider
Session::extend('datastore', function ($app) {
    return new \Illuminate\Session\Handlers\... // wrap or implement SessionHandlerInterface
});

DatastoreSessionHandler implements SessionHandlerInterface, so registering it via Session::extend() and setting 'driver' => 'datastore' in config/session.php works — but you must pass session.gc_maxlifetime through to its constructor or expired sessions never get swept.

Testing with the emulator. Point DATASTORE_EMULATOR_HOST at a container in CI, and wipe namespaces between tests:

protected function setUp(): void
{
    parent::setUp();
    $this->ds = new DatastoreClient([
        'projectId'   => 'test',
        'namespaceId' => 'test-' . Str::random(8),
    ]);
}

A fresh random namespace per test is far cheaper than trying to delete every entity.

Gotchas and Tips

Empty arrays and zero-value properties disappear. Datastore does not persist a property set to [], '', 0, or false if the value is "empty" in ways the serializer treats as unset. Always read with ?? defaults or coalesce in your model accessors. This is the single most common "my field vanished" bug report.

lookup() returns null, never throws. Missing entities are indistinguishable from a successful miss — no exception to catch. lookupBatch() returns an array where missing keys are null at the same index, so always check.

Keys are immutable. You cannot rename an entity's key or change its parent. "Move" means lookup → insert under the new key → delete old — and there is no atomicity across that unless both ops share a parent. Model your keys so they never need to change.

Queries without an ancestor are eventually consistent. A write followed immediately by a filtered query may not see the write. If a UI does "create then redirect to list", either query by key (strongly consistent), use an ancestor query, or read back the specific key rather than the list.

Composite indexes are deployed, not inferred. Unlike MongoDB, Datastore will fail a query that needs an unbuilt index with gRPC code 9 (FAILED_PRECONDITION). The exception message contains a gcloud datastore indexes create snippet — parse it, paste into index.yaml, and deploy via your CI pipeline. Treat index.yaml as schema and review it in PRs. Index builds are asynchronous and can take minutes on large datasets, so ship index changes before the code that uses them.

Beware index explosion on long strings and embedded entities. Every indexed property costs index entries, and a single entity caps at 20,000. Store large blobs, JSON payloads, and markdown bodies with excludeFromIndexes:

$datastore->entity($key, $data, ['excludeFromIndexes' => ['body', 'rawPayload']]);

Indexed strings truncate at ~1,500 bytes and lose the tail for filtering — don't rely on = matching on long text.

DateTime objects store UTC microseconds and lose the timezone. Datastore has no timezone-aware type. Convert on the way in and on the way out; otherwise a user in Europe/Paris sees their timestamps silently drift. Also, Entity returns a DateTime instance, not a Carbon — wrap it if you want diffForHumans().

Binary data needs Blob. A raw PHP string is stored as a UTF-8 string and will corrupt on binary. Wrap with new Blob($bytes).

Batch limits are hard: 500 entities per commit, 10 MiB per commit, 1 MiB per entity, ~1 write/sec per entity group. Exceeding these throws INVALID_ARGUMENT, not a soft warning.

The Entity object is not an array. json_encode($entity) works but produces a lossy representation (DateTime → string, Key → nested object). Call $entity->get() to get a plain array, and be explicit in API resources.

Config caching bites env() calls. If you lazily construct DatastoreClient inside a closure that reads env('GOOGLE_CLOUD_PROJECT'), it breaks after php artisan config:cache. Always route through config('datastore.*').

Octane and long-lived workers: bind the client as a singleton and give it a finite requestTimeout (e.g. ['requestTimeout' => 10]). A hung gRPC call in a worker stalls the whole request. Also flush any cached per-namespace clients in a RequestTerminated listener if you switch tenants per request.

Error handling surface: all failures are Google\Cloud\Core\Exception\ServiceException (or subclasses like NotFoundException, BadRequestException). The gRPC status code is on getCode() — 4 = deadline exceeded (retry), 8 = quota exhausted (back off), 9 = missing index (fix index.yaml), 10 = ABORTED (transaction contention, retry). Wrap writes in retry() with exponential backoff for codes 4, 10, and 14.

Don't use it as a queue. There's no native SELECT ... FOR UPDATE SKIP LOCKED and no cheap polling. Use Cloud Tasks or Pub/Sub for job dispatch, and Datastore only for durable job state.

Cost model favors few large reads over many small ones. Reads are billed per entity (with a small-entity tier under 1 KiB), writes per entity. Design denormalized documents that answer a page in one query rather than fanning out to 20 lookups.

Firestore in Datastore mode is the successor. New GCP projects provision Firestore in Datastore mode; the API surface is identical and this client works unchanged. If you're greenfield and don't need the Datastore-mode quirks, evaluate the google/cloud-firestore client — it has stronger consistency, real-time listeners, and more active development (note the 25-star count on this repo; it's a stable-but-quiet library).

Cloud Datastore Laravel Package

Getting Started

Implementation Patterns

Gotchas and Tips