Queue Sql Laravel Package

Getting Started

Minimal path from install to a useful job:

1. composer require kamranata/queue-sql and publish config (php artisan vendor:publish --tag=queue-sql-config if a config tag is offered) — check config/queue-sql.php for the default connection/queue, chunk size, and maxJobs cap before dispatching anything real.
2. Make sure a worker is actually running on the queue you're dispatching to (php artisan queue:work --queue=...). Nothing in this package executes inline; dispatch() only enqueues.
3. First use case: replace a synchronous bulk write that's timing out or blocking a request.

// Before: one long request, no retry story
DB::table('stats')->insert($rows);

// After: enqueued, chunked, retryable
DB::table('stats')->queue(chunk: 1000)
    ->insert($rows)
    ->dispatch();

Then confirm the fan-out landed correctly with the status command — it is the primary debugging surface and the first thing to reach for when a job "did nothing":

php artisan queue-sql:status
php artisan queue-sql:status --json          # new in 1.4.0
php artisan queue-sql:status --watch         # new in 1.4.0, TTY only

Where to look first: the terminal methods on the query builder (insert, and now upsert), the queue() builder options (chunk, maxJobs), and queue-sql:status. Those three cover ~90% of day-to-day use.

Implementation Patterns

Append-only bulk writes → insert(). Keep this for logs, events, and anything where a duplicate row is harmless or the source data is guaranteed unique. It is the faster path and has the simplest failure semantics.

Re-runnable imports / retried jobs → upsert() (new in 1.4.0). This is the headline feature. The fan-out shape mirrors insert(), but each chunk is an idempotent upsert, so a job that retries after a partial failure cannot duplicate rows.

DB::table('stats')->queue(chunk: 1000, maxJobs: 20)
    ->upsert(
        $rows,
        uniqueBy: ['user_id', 'day'],
        update:   ['views'],
    )
    ->dispatch();

Use it for scheduled syncs, webhook replays, CSV imports the user can re-trigger, and anything sitting behind tries > 1. Keep update: narrow — listing only the columns that should actually change prevents a retry from clobbering columns another process wrote in the meantime.

Pair upsert() with a real unique index. uniqueBy is only advisory depending on driver (see Gotchas). Add the composite unique index in a migration before shipping the job; without it the upsert degrades into an insert and the idempotency guarantee is gone.

Chunk sizing. chunk is rows per job, not necessarily rows per statement. Drive the number off your placeholder budget and row width — wide tables want smaller chunks. If maxJobs is set, a very large input silently splits into more work than the cap allows; check status output for whether everything was actually enqueued before assuming the write is complete.

Monitoring. --json is the integration point:

php artisan queue-sql:status --json | jq '.batches[] | select(.state != "finished")'

Feed it to a scheduled health check or a dashboard. Batch names carry the operation and table (queue-sql:upsert:{table}), so you can filter by table when several imports run concurrently. If Horizon is installed, the same tags show up there — prefer Horizon for per-job inspection and --json for aggregate state.

--watch for humans, not for machines. Use it while tailing a backfill locally or in a deploy window. It refreshes every 2s by default (--interval=N to change).

Grouping work. Dispatch related imports in the same request/job so batch names correlate, and be deliberate about afterCommit / Bus::batch if the dispatch happens inside a transaction — otherwise workers can pick up rows that the transaction later rolls back.

Gotchas and Tips

• upsert idempotency is per row, not per column-set. A retried chunk re-applies update: columns with the values from that chunk's payload. If the payload is stale relative to the DB, last-write-wins can move data backwards. Only include columns your job is authoritative for.
• uniqueBy is honoured differently per driver. MySQL's ON DUPLICATE KEY UPDATE keys off any unique index, not the columns you passed; Postgres/SQLite use your list. A second, unrelated unique constraint on the table will cause rows to update when you expected an insert. Test the upsert on the actual production driver, not just SQLite in CI.
• Query-builder upserts don't manage timestamps. created_at / updated_at and soft-delete columns are yours to set in the payload. A common post-migration surprise is updated_at silently not moving because update: ['views'] didn't include it.
• --watch renders once when not on a TTY. Piped, redirected, or cron-invoked runs don't loop. Don't wire --watch into a supervisor or CI step expecting live output — use --json and poll externally.
• Treat the --json shape as an interface. It's new in 1.4.0 and will likely grow fields. Parse defensively (select keys, don't index positionally) if a dashboard depends on it.
• dispatch() returns before the data exists. Any read-your-writes expectation immediately after dispatch will fail. If downstream code needs the rows, chain on the batch/job completion callback rather than the dispatch call.
• Retries are only safe with upsert(). A failed insert() chunk that retries will duplicate whatever rows landed before the failure. If a job has tries > 1 and the table lacks a unique constraint, that's a latent duplicate-row bug — this release is a good excuse to audit those.
• Diagnosing "nothing happened": check worker is running on the right queue → queue-sql:status --json for batch presence → the queue-sql:upsert:{table} tag in Horizon for individual failures. Nine times out of ten it's a queue mismatch or maxJobs capping the fan-out, not a package bug.
• Extension point worth knowing: the batch naming convention (queue-sql:{operation}:{table}) is stable enough to key off in your own monitoring or alerting; wrap it in one helper rather than string-matching queue-sql: across the codebase.