Meilisearch index synchronization can run concurrently on multiple API replicas

[Falenos]

What happened?

When two or more API replicas start against the same MongoDB and Meilisearch instances, more than one replica can execute api/db/indexSync.js concurrently.

indexSync() coordinates through FlowStateManager.createFlowWithHandler(). That method performs a get, waits 250 ms, performs another get, and then writes the initial PENDING state. The final claim is not atomic, so replicas starting together can all observe no flow and execute the handler.

The handler can update shared Meilisearch settings, reset MongoDB _meiliIndex flags, clean indexes, and start full message/conversation indexing. Concurrent handlers therefore duplicate expensive work and can reset progress while another replica is indexing.

Expected: only one API replica performs the cluster-wide Meilisearch reconciliation.

Actual: synchronized replicas can all enter the reconciliation handler.

Version Information

Confirmed in current dev commit 8f6228095fcf118ebe4ee28fd960638f0820704b. The relevant non-atomic claim is present in packages/api/src/flow/manager.ts, and api/db/indexSync.js uses it as the concurrency guard.

Steps to Reproduce

1. Configure two API replicas with SEARCH=true, sharing MongoDB, Redis, and Meilisearch.
2. Ensure message or conversation synchronization is required (for example, enough documents have _meiliIndex != true, or index settings require an update).
3. Start both API replicas at approximately the same time.
4. Observe both replicas enter createFlowWithHandler() before either initial state write becomes visible.
5. Observe duplicate [indexSync] Starting index synchronization check... / sync-start log sequences and concurrent writes to the shared index state.

A deterministic unit reproduction can use two FlowStateManager instances over the same delayed store and pause both calls after their second get; both proceed to set and execute their handlers.

What browsers are you seeing the problem on?

No response — backend concurrency issue.

Relevant log output

replica-a [indexSync] Starting index synchronization check...
replica-b [indexSync] Starting index synchronization check...
replica-a [indexSync] Messages need syncing: 3/95000 indexed
replica-b [indexSync] Messages need syncing: 3/95000 indexed
replica-a [indexSync] Starting message sync (94997 unindexed)
replica-b [indexSync] Starting message sync (94997 unindexed)

Code of Conduct

I agree to follow this project's Code of Conduct.

Proposed fix

The fix is implemented in #15465.

[gyanu2507]

The guard in FlowStateManager.createFlowWithHandler() is not a lock. A get, 250 ms sleep, second get, then write PENDING is a classic TOCTOU. Two replicas starting together both see no flow and both run indexSync().

That handler can rewrite Meilisearch settings, flip _meiliIndex back to false, and kick off a full message/conversation reindex. Two pods doing that on ~95k documents is how you get duplicate work and a reset of progress the other replica already made.

Claim the job with something atomic, then run the handler:

• Redis: SET index-sync:lock <replica> NX EX 3600 (or whatever TTL covers a worst-case sync)
• Mongo: findAndModify / findOneAndUpdate with status: PENDING only when missing or expired

createFlowWithHandler() should use that primitive for cluster-wide work. Per-request flows can stay as they are. Expected behavior is what you wrote: one replica owns Meilisearch reconciliation for the cluster.

[Falenos]

Thanks — agreed on the TOCTOU analysis. #15465 now puts an atomic MongoDB coordinator around the complete indexSync operation before FlowStateManager is entered. The coordinator claims the fixed _id with findOneAndUpdate(..., { upsert: true }); simultaneous first claims are serialized by MongoDB’s unique _id, and the duplicate-key loser becomes a follower rather than executing the handler. The owner holds a renewable, owner-qualified lease, and renewal, completion, and failure transitions all require the same owner and an unexpired expiresAt. Followers wait for a valid completion marker or take over only after the lease/failure marker expires. We intentionally leave FlowStateManager unchanged for its existing per-request use cases, as you suggested. The coordinator tests use mongodb-memory-server and run two real coordinators against the same collection to prove only one handler executes, with additional expired-takeover and stale-owner fencing coverage.

[jayblast-spec]

The NX-lock direction is right, but I'd reach for Mongo before Redis here, since Mongo is the one store both replicas are guaranteed to share (LibreChat needs it to run at all; Redis is only in the picture if you've configured it as the cache/pubsub backend). You can get the atomic claim out of the same collection FlowStateManager already writes to, in one round trip:

const claimed = await FlowState.findOneAndUpdate(
  {
    flowId,
    $or: [
      { status: { $exists: false } },
      { status: 'FAILED' },
      { status: 'PENDING', lastHeartbeat: { $lt: new Date(Date.now() - STALE_MS) } },
    ],
  },
  { $set: { status: 'PENDING', owner: replicaId, lastHeartbeat: new Date() } },
  { upsert: true, returnDocument: 'after' }
);
if (claimed.owner !== replicaId) return; // someone else already owns this flow

That collapses the get, sleep, get, set into a single conditional write, so there's no window where two replicas both observe "no flow." Whoever loses the findOneAndUpdate just returns.

The part neither the original report nor a plain TTL lock handles well is that this specific job can run long, 95k messages in the log you posted. A lock TTL picked once at claim time is wrong either direction: long enough to survive a real full sync, and a crashed replica's lock lingers for the whole window before anyone can retry; short enough to fail over fast, and a slow-but-healthy sync gets preempted mid-run by a second replica that thinks the first one died.

The fix is to stop treating staleness as "time since claim" and instead track "time since last heartbeat." The owning replica updates lastHeartbeat on its own claim document periodically while it's actually making progress (say every sync batch, or every N seconds), and the staleness check in the query above compares against that instead of a fixed expiry from the start. A replica that's alive and working keeps refreshing its own claim indefinitely; one that crashes stops heartbeating and becomes reclaimable within one stale-check interval, regardless of how far into the 95k it got. That's the same pattern as a lease in any job queue, the lease length only needs to cover the gap between heartbeats, not the whole job.

[Falenos]

Thanks, @jayblast-spec — the distinction between a fixed whole-job TTL and a renewable liveness deadline was exactly the right one, and it prompted a useful tightening of #15465.

The updated implementation now:

• claims a fixed MongoDB job document with one conditional findOneAndUpdate; losing replicas remain followers;
• renews an owner-qualified expiresAt every minute, with a three-minute stale-owner window;
• fences renewal, completion, and failure by both owner and an unexpired lease;
• allows a healthy reconciliation to renew for as long as needed rather than imposing a whole-job timeout;
• includes real-Mongo regressions for contention, healthy renewal, missed-heartbeat takeover, and stale-owner finalization.

One implementation detail differs from the example: FlowStateManager is Keyv-backed and may use Redis or process-local memory, since Redis is optional. There is therefore no Mongo-backed FlowState collection guaranteed to be shared by every replica. The PR uses a small dedicated distributedJobs collection because MongoDB is the mandatory shared backend.

Current dev also made FlowState claims atomic, but retaining FlowState inside the Mongo lease would create two independent coordinators: after reclaiming an expired Mongo lease, a replacement owner could still wait on the previous owner's pending FlowState. The nested FlowState layer was therefore removed so MongoDB remains the single ownership authority for this job.

Renewal is time-based rather than batch-only because an owner may be waiting on a bounded MongoDB or Meilisearch operation that can still mutate shared state. Expiring it during that wait could permit overlap. Thanks again for calling out the failover tradeoff so clearly.