Today's Redis search works. It runs at 100% in 12 regions and returns the right drivers in about 21 ms on average. Its cost is reads: it fetches every driver in the radius before it filters, so reads grow with driver density. In Metro Manila at 10%, that pushed Redis to 35–59K ops/s against a 25K plan. The index returns the same drivers from a few reads.
1 GEOSEARCH, then HGETALL driver:{uid} for every driver in the radius, available or not.
Production average, Metro Manila with the legacy flag at 10%.
✓ Nearest 5 drivers
GEOSEARCH … COUNT n, then HMGET T:card for the nearest drivers only.
Prototype average. Offline drivers, drivers on a trip and drivers of other services are not in the set.
✓ Same nearest 5 drivers
redissynclocations) and MOVE (updateLocation) write it. The new reconciler (driverIndexReconciler) repairs it and writes the heartbeat. Every Gen 2 search function reads it.firestore-sync service, and the Gen 1 qr and v1 functions.gcp-redis-driver-search-indexHGET T:meta reconciledAtindex_stale.T:geo:<svc> … COUNT nT:card, then the same JS filtersparseDriverHash and matchesFilters run unchanged, then the first limit drivers are kept.limit drivers passed, n grows by at least 1.5× and the search repeats steps 3 and 4. It stops when limit drivers pass or the reply is short.When the index cannot answer, the fallback flag gcp-redis-driver-search-firestore-fallback decides what search returns. Production and staging use always. This table applies in index mode on. In shadow the caller always gets today's result.
| Reason | always | error_only | none |
|---|---|---|---|
| Index answered | index | index | index |
index_stale: write flag off, or heartbeat old, missing or unreadable | Firestore | Firestore | Firestore |
redis_error: GEOSEARCH or HMGET failed | Firestore | Firestore | empty |
no_candidates: no driver in the radius | Firestore | empty | empty |
all_filtered: drivers found, none passed | Firestore | empty | empty |
A region the reconciler does not know has no heartbeat, so it reads Firestore. The fallback after Firestore is turned off is in Rollout.
Each GEOSEARCH asks for one page of n drivers. If too few of them pass the filters, the search reads a bigger page and repeats until it has limit drivers or the radius has no more. The index holds only drivers who can take a trip, so the result is the same as reading every driver in the radius.
The page keeps a typical search small. That matters for GET /nearbyDrivers, whose radius comes from the request with no upper limit.
Data per search in Metro Manila, estimated. Without COUNT, one wide GET /nearbyDrivers call reads the whole region, about 2.9 MB. With COUNT it stops after one page when the filters pass.
All keys of one region share one hash tag, so a Lua script can touch several of them on a Redis cluster.
T = {dsidx:v1:<encodeURIComponent(region)>} region = the d.region value
T:geo:any GEO uid → lng,lat online, status AVAILABLE (0), region R, position known
T:geo:<svc> GEO the same set, limited to drivers who offer svc
svc: mctaxi delivery pabili tricycle hmdelivery car taxicab (7 sets)
T:card HASH uid → JSON { v, m, w, p, f }
T:meta HASH reconciledAt, eligible, members, drift, cards (reconciler)
writerAt (APPLY from redissynclocations)
moverAt (MOVE from updateLocation)
updateTime in microseconds, 16 digits zero-padded. APPLY rejects an older version.
wThe set mask the state wants. 0 means the driver is not eligible.
mThe set mask applied. It stays 0 until a position is known.
pThe last position: { t, lat, lng, f }, with t the ping time in ms.
fThe search fields, string-encoded like driver:{uid}, so the existing parsers read them unchanged.
Set membership encodes only per-driver state: online, status, region and services. Seats, currentregion and required fields stay in the existing JS filters on the card.
An extra member costs a few bytes. A missing member would change results, so every rule leans toward including a driver. A driver who leaves keeps a 43-byte tombstone card, { v, m: 0, w: 0 }, so a late, older write is rejected.
The index read flag takes precedence over the legacy flag.
Once a region runs the index at 100%, its legacy flag goes off. The legacy search code is deleted after every region is on the index.
gcp-redis-driver-search decides where each region searches today.
In staging the legacy flag is on for every region.
Both new flags go into flagsDefaultFalseMapping in the LaunchDarkly utils.
| Flag | Values | Default | Context | Controls |
|---|---|---|---|---|
gcp-redis-driver-search-index (new) | off · shadow · on | off | user (rider key and service), region | The index read path |
gcp-redis-driver-index (new) | true · false | false | user gcp_serverless_v1, region | Every index write, the stream loop included |
gcp-redis-driver-search-firestore-fallback | always · error_only · none | always | user, region | Firestore fallback. Read only in index mode on. |
enable-dr-2780-driver-search-logging | true · false | false | user, region | The Driver search source and Driver search shadow log lines |
gcp-redis-driver-search | off · shadow · on | off | user, region | The legacy Redis search. Read only when the index mode is off or shadow. |
The user context gets a service attribute set to K_SERVICE, the function's name. GET /nearbyDrivers and the assignment loops both pass a rider key, so a rule on user.service is what lets the assignment loops move to the index first while qrgen2 and v1gen2 stay off. Each can be rolled back on its own.
Every Gen 2 search function reads the index. Gen 1 runs the same search code but stops at a guard, because it has no Redis URL.
On master, qrGen2 and v1Gen2 run a generated copy of the Gen 1 search file. On new-staging they import joyride-utils/driver-search, so preprod never runs the generated copy. Three checks on the release PR cover that gap. Only the read code is copied. The write code lives only in utils.
utils/driver-search/index.js and the Gen 1 file functions/api/utils/driverSearch.util.js. They differ only in three require lines and one guard.helpers/driverSearch.util.js and the extraction.json ledgers with analyze() from scripts/gen2-packages/source-slices.cjs.require lines and the guard are normalized.bash scripts/gen2-packages/linux.sh passes before merge, because no PR pipeline runs that drift check.deploy.sh writes only GCP_REDIS_URL for qrGen2 and v1Gen2. If production Redis runs in cluster mode, it also needs GCP_REDIS_CLUSTER_ENABLED and GCP_REDIS_CLUSTER_NODES. That is checked before these two functions read the index.Every write checks gcp-redis-driver-index for its region first. False or unreadable: the write does nothing. No write throws to its caller.
updateLocation
T:geoT:cardmoverAtredissynclocations
locations writeT:geoT:cardwriterAtdriverIndexReconciler
T:geoT:cardreconciledAtreconcilePositions
POSITION_INTERVAL_MSdriver:{uid}→ MOVEupdateLocation)w is 0. A late ping cannot bring back an offline driver.moverAt on every call, including calls that change nothing.redissynclocations)locations write.writerAt.driverIndexReconciler)T:meta.firestore-sync already writes positions to GCP Redis but does not call MOVE.reconcilePositions always runs. It lists the index members, reads their positions from driver:{uid}, and runs MOVE where the hash is newer. It runs on its own interval, POSITION_INTERVAL_MS. 0 turns it off.location:firestore-writeback:updates on its own connection and moves AWS drivers within about 2 s.The reconciler writes the heartbeat only while every condition holds. Search reads the index only while the heartbeat is fresh.
writerAt less than 180 s oldmoverAt less than 180 s oldA missed deploy never breaks search. Each case ends on today's code or on Firestore.
| Not redeployed | Effect | Search reads |
|---|---|---|
A joyride-utils search caller | Runs the old search code, legacy flag only | same as today |
qrGen2 or v1Gen2 | Runs the old generated copy, legacy flag only | same as today |
Gen 1 api, qr, v1 | Never needs a deploy. The guard returns off. | Firestore |
redissynclocations | No status writes to the index. writerAt goes stale and the heartbeat stops. | Firestore |
updateLocation | No MOVE. moverAt goes stale and the heartbeat stops. | Firestore |
| Reconciler | No heartbeat and no stream loop. Search sees index_stale. | Firestore |
Same as today means Firestore in Metro Manila and the legacy Redis search in the legacy regions.
The first bar is the incident peak, with Metro Manila on the legacy search at 10%. The other bars are estimates from prototype costs and today's rates. Worst case counts every command inside a Lua script as one billed operation.
| Bar | Baseline | Search | MOVE | APPLY | Repair pass | Positions | Total |
|---|---|---|---|---|---|---|---|
| Incident peak, Metro Manila at 10% | legacy search reads | 59K | |||||
| Typical | 3.3K | 2.28K | 1K | 0.1K | 0.15K | 0 | 6.8K |
| Worst case, positions every 60 s | 3.3K | 2.28K | 11K | 1.1K | 0.15K | 0.07K | 17.9K |
| Worst case, positions every 15 s | 3.3K | 2.28K | 11K | 1.1K | 0.15K | 0.29K | 18.1K |
Search is 950 searches per second × 2.3 commands, plus at most 0.08K for searches that read to the end of the radius.
New load may use at most 70% of the room the plan has left: 0.7 × (25K − 3.3K) = 15.2K ops/s, so the total stays under 18.5K.
In the worst case the new load uses 14.6K to 14.8K of the 15.2K. That leaves room for about 165 to 260 searches per second from qrGen2 and v1Gen2. Their search rate is not known yet, so they move last, after their own check.
How Redis Cloud bills the commands inside a Lua script is checked in preprod before any production flag moves. In the typical accounting the room left is about 11.7K ops/s.
cardsRead.
Hot slotThe prototype ran 35.4K ops/s at 0.53 CPU on the busiest slot. Production load is about a third of that.
FirestoreAbout 6.2M reads per day for the Metro Manila repair pass, about $3.7 per day
At any step, one flag change rolls back: set the read flag gcp-redis-driver-search-index to off, or the write flag gcp-redis-driver-index to false. With both new flags off, every deploy changes nothing.
new-stagingqrGen2 and v1Gen2 included, then turn on index writes.qrGen2 and v1Gen2 included.cardsRead per search: p99 at most 500, maximum at most the region's eligible drivers. At most 5 commands per search on a dense test set.GET /nearbyDrivers and one QR booking. The Driver search source log shows source: redis with the right user.service.masterqrGen2, v1Gen2ZCARD drivers:geo, the status-change rate and Lua billing. The PR must show new load at or below 15.2K, or the rollout stops.updateLocation, redissynclocations, onTripCreateV2, onTripUpdateV2, engine_queue_onUpdateV2 (each shard), qrGen2, v1Gen2. Gate: the expected DD_VERSION on each function, the legacy flag unchanged, no {dsidx key in Redis.gcp-redis-driver-index true for Metro Manila. The reconcile log shows low drift and a steady heartbeat.shadow in Metro Manila, except qrgen2 and v1gen2. Gate over at least 2,000 searches: 0 mismatches, cardsRead p99 at most 500, tail network out at most 5 MB/s.qrGen2 and v1Gen2. First check the cluster keys, their search rate and their peak instances against the Redis connection limit. v1Gen2 allows up to 1,000 instances at concurrency 1. Then shadow, then on from 1% to 100%.13 open. Each must be answered before the step named after it.
updateTime? Unit test, then one preprod trigger.region_configs equal the driver's region value? Compare the values.drivers:geo?K_SERVICE value in each search function. From S7, needed before P3.qrGen2 and v1Gen2 search rate and peak instances. Before P6.Acceptance: with the index on, riders get the same drivers as today.
w = 0 and bad positions leave the index correct. A storm test checks that no eligible driver goes missing.reconcilePositions moves a driver only when the hash is newer, and interval 0 turns it off. The stream loop moves AWS drivers, skips other entries, keeps the newest entry per driver, and a stalled loop stops the heartbeat.GCP_REDIS_URL, the index flag is never read.service = K_SERVICE.GET /nearbyDrivers and one QR booking with the index on.release/gcp-redis-index from master, PR to master. feature/gcp-redis-index has PRs to it.feature/gcp-redis-index-staging from new-staging, with the same commits cherry-picked.driverSearch.util.js change, the re-slice and the sync test go to master only. new-staging has no Gen 1 driverSearch.util.js.Only the locations collection stops being written and read. Trips, region_configs and other config stay in Firestore. Three phases after P6 each have their own PR, gate and rollback.
Removes: nothing yet
Gate: no Driver state not direct-written line for 7 days
Removes: the write feed
Gate: writer list complete, clock skew under 500 ms, version source picked
Removes: the primary read and the fallback read
Gate: auto-accept rate known, atomic claim, recovery plan after a Redis flush
setDriverState(uid, patch), writes driver:{uid}. Every state writer calls it after its Firestore write. The trigger still feeds the index.Driver state not direct-written when it sees a change the helper did not write.redissynclocations skips both.gcp-redis-driver-state-source switches a region between trigger and direct. Rollback: set it back to trigger.drivers:geo every 300 s.TIME or a version stamped by the writer.index_stale no longer forces Firestore, and the geofirestore query builders are deleted..near( query on locations is left in the Gen 2 functions.| Reason | Result |
|---|---|
| Index answered | index |
index_stale | index, logged as stale |
redis_error, no_candidates, all_filtered | empty |
A Datadog monitor watches heartbeat age. A stale index is better than an empty result, because by then the index is the only driver data left.
drivers:geoThat is the path that caused the incident: about 349 HGETALL calls per search. A region-wide fallback to it would repeat the incident.
The AWS writer already writes GCP Redis, and reconcilePositions reads driver:{uid}. Phase F needs no AWS change.