Today one search reads the record of every driver in drivers:geo inside the radius, and nothing ever removes a driver from that set. The index holds only drivers who can take a trip now, so a search reads a few nearby drivers in about 2 commands.
1 GEOSEARCH, then HGETALL driver:{uid} for every driver in the radius, available or not.
Average in Metro Manila with the legacy flag at 10%. All of them share one 300 ms timeout.
GEOSEARCH … COUNT n, then HMGET T:card for the nearest drivers only.
Offline drivers, drivers on a trip and drivers of other services are not in the set.
gcp-redis-driver-search-index to off and search runs today's code.region value, for example Metro Manila. Each region has its own index and its own flag rules.onTripCreateV2, onTripUpdateV2, engine_queue_onUpdateV2, qrGen2 and v1Gen2, which serves v1Application.qr and v1 functions. They have no GCP Redis access, stay on Firestore and do not change.on, shadow, offon serves index results. shadow serves today's results and compares them with the index. off runs today's code.Moving dots show data flow. Grey keys are the existing GCP Redis keys, which do not change. Every writer here writes GCP Redis.
gcp-redis-driver-search-indexoff runs today's code. shadow serves today's result and compares. on uses the index and does not read the legacy flag.HGET T:meta reconciledAtindex_stale.T:geo:<svc> … COUNT nT:card, then the same JS filtersparseDriverHash and matchesFilters run unchanged, then the first limit drivers are kept.limit drivers passed, n grows by at least 1.5× and the search repeats steps 3 and 4. It stops when limit drivers pass or the reply is short.When the index cannot answer, the fallback flag gcp-redis-driver-search-firestore-fallback decides what search returns. Production and staging use always. This table applies in index mode on. In shadow the caller always gets today's result.
| Reason | always | error_only | none |
|---|---|---|---|
| Index answered | index | index | index |
index_stale: write flag off, or heartbeat old, missing or unreadable | Firestore | Firestore | Firestore |
redis_error: GEOSEARCH or HMGET failed | Firestore | Firestore | empty |
no_candidates: no driver in the radius | Firestore | empty | empty |
all_filtered: drivers found, none passed | Firestore | empty | empty |
A region the reconciler does not know has no heartbeat, so it reads Firestore. The fallback after Firestore is turned off is in Rollout.
Each GEOSEARCH asks for one page of n drivers. If too few of them pass the filters, the search reads a bigger page and repeats until it has limit drivers or the radius has no more. The index holds only drivers who can take a trip, so the result is the same as reading every driver in the radius.
The page keeps a typical search small. That matters for GET /nearbyDrivers, whose radius comes from the request with no upper limit.
Data per search in Metro Manila, estimated. Without COUNT, one wide GET /nearbyDrivers call reads the whole region, about 2.9 MB. With COUNT it stops after one page when the filters pass.
All keys of one region share one hash tag, so a Lua script can touch several of them on a Redis cluster.
T = {dsidx:v1:<encodeURIComponent(region)>} region = the d.region value
T:geo:any GEO uid → lng,lat online, status AVAILABLE (0), region R, position known
T:geo:<svc> GEO the same set, limited to drivers who offer svc
svc: mctaxi delivery pabili tricycle hmdelivery car taxicab (7 sets)
T:card HASH uid → JSON { v, m, w, p, f }
T:meta HASH reconciledAt, eligible, members, drift, cards (reconciler)
writerAt (APPLY from redissynclocations)
moverAt (MOVE from updateLocation)
updateTime in microseconds, 16 digits zero-padded. APPLY rejects an older version.
wThe set mask the state wants. 0 means the driver is not eligible.
mThe set mask applied. It stays 0 until a position is known.
pThe last position: { t, lat, lng, f }, with t the ping time in ms.
fThe search fields, string-encoded like driver:{uid}, so the existing parsers read them unchanged.
Set membership encodes only per-driver state: online, status, region and services. Seats, currentregion and required fields stay in the existing JS filters on the card.
An extra member costs a few bytes. A missing member would change results, so every rule leans toward including a driver. A driver who leaves keeps a 43-byte tombstone card, { v, m: 0, w: 0 }, so a late, older write is rejected.
The index read flag takes precedence over the legacy flag.
Once a region runs the index at 100%, its legacy flag goes off. The legacy search code is deleted after every region is on the index.
gcp-redis-driver-search decides where each region searches today.
In staging the legacy flag is on for every region. The fallback flag is always in production and staging.
Both new flags go into flagsDefaultFalseMapping in the LaunchDarkly utils.
| Flag | Values | Default | Context | Controls |
|---|---|---|---|---|
gcp-redis-driver-search-index (new) | off · shadow · on | off | user (rider key and service), region | The index read path |
gcp-redis-driver-index (new) | true · false | false | user gcp_serverless_v1, region | Every index write, the stream loop included |
gcp-redis-driver-search-firestore-fallback | always · error_only · none | always | user, region | Firestore fallback. Read only in index mode on. |
enable-dr-2780-driver-search-logging | true · false | false | user, region | The Driver search source and Driver search shadow log lines |
gcp-redis-driver-search | off · shadow · on | off | user, region | The legacy Redis search. Read only when the index mode is off or shadow. |
The user context gets a service attribute set to K_SERVICE, the function's name. GET /nearbyDrivers and the assignment loops both pass a rider key, so a rule on user.service is what lets the assignment loops move to the index first while qrgen2 and v1gen2 stay off. Each can be rolled back on its own.
Six Gen 2 functions search for drivers, and all six read the index. Gen 1 runs the same search code but stops at a guard, because it has no Redis URL.
On master, qrGen2 and v1Gen2 run a generated copy of the Gen 1 search file. On new-staging they import joyride-utils/driver-search, so preprod never runs the generated copy. Three checks on the release PR cover that gap. Only the read code is copied. The write code lives only in utils.
utils/driver-search/index.js and the Gen 1 file functions/api/utils/driverSearch.util.js. They differ only in three require lines and one guard.helpers/driverSearch.util.js and the extraction.json ledgers with analyze() from scripts/gen2-packages/source-slices.cjs.require lines and the guard are normalized.bash scripts/gen2-packages/linux.sh passes before merge, because no PR pipeline runs that drift check.deploy.sh writes only GCP_REDIS_URL for qrGen2 and v1Gen2. If production Redis runs in cluster mode, it also needs GCP_REDIS_CLUSTER_ENABLED and GCP_REDIS_CLUSTER_NODES. That is checked before step P6.Every write first checks gcp-redis-driver-index for its region. If the flag is false or cannot be read, the write does nothing. No write ever throws an error to its caller.
redissynclocations, after each Firestore locations write.writerAt.updateLocation, after the position is written.w is 0. A late ping cannot bring back an offline driver.moverAt on every call, including calls that change nothing.driverIndexReconciler, a worker pool with one instance. One pass, then it sleeps for the rest of 60 s.T:meta.firestore-sync already writes positions to GCP Redis but does not call MOVE. Nothing changes on the AWS side.reconcilePositions always runs. It lists the index members, reads their positions from driver:{uid}, and runs MOVE where the hash is newer. It runs on its own interval, POSITION_INTERVAL_MS. 0 turns it off.location:firestore-writeback:updates on its own connection and moves AWS drivers within about 2 s.Turn parts off. The reconciler writes the heartbeat only if every condition holds, and search reads the index only while the heartbeat is fresh.
writerAt less than 180 s oldmoverAt less than 180 s oldA missed deploy never breaks search. Each case ends on today's search path.
| Not redeployed | Effect | Search reads |
|---|---|---|
A joyride-utils search caller | Runs the old search code, legacy flag only | same as today |
qrGen2 or v1Gen2 | Runs the old generated copy, legacy flag only | same as today |
Gen 1 api, qr, v1 | Never needs a deploy. The guard returns off. | Firestore |
redissynclocations | No status writes to the index. writerAt goes stale and the heartbeat stops. | Firestore |
updateLocation | No MOVE. moverAt goes stale and the heartbeat stops. | Firestore |
| Reconciler | No heartbeat and no stream loop. Search sees index_stale. | Firestore |
Same as today means Firestore in Metro Manila and the legacy Redis search in the legacy regions.
The first bar is the incident peak, with Metro Manila on the legacy search at 10%. The other bars are estimates from prototype costs and today's rates. Worst case counts every command inside a Lua script as one billed operation. Hover a segment for its value.
| Bar | Baseline | Search | MOVE | APPLY | Repair pass | Positions | Total |
|---|---|---|---|---|---|---|---|
| Incident peak, Metro Manila at 10% | legacy search reads | 59K | |||||
| Typical | 3.3K | 2.28K | 1K | 0.1K | 0.15K | 0 | 6.8K |
| Worst case, positions every 60 s | 3.3K | 2.28K | 11K | 1.1K | 0.15K | 0.07K | 17.9K |
| Worst case, positions every 15 s | 3.3K | 2.28K | 11K | 1.1K | 0.15K | 0.29K | 18.1K |
Search is 950 searches per second × 2.3 commands, plus at most 0.08K for searches that read to the end of the radius.
New load may use at most 70% of the room the plan has left: 0.7 × (25K − 3.3K) = 15.2K ops/s, so the total stays under 18.5K.
In the worst case the new load uses 14.6K to 14.8K of the 15.2K. That leaves room for about 165 to 260 searches per second from qrGen2 and v1Gen2. Their search rate is not known yet, so they move last, after their own check.
How Redis Cloud bills the commands inside a Lua script is checked in preprod before any production flag moves. In the typical accounting the room left is about 11.7K ops/s.
cardsRead.
Hot slotThe prototype ran 35.4K ops/s at 0.53 CPU on the busiest slot. Production load is about a third of that.
FirestoreAbout 6.2M reads per day for the Metro Manila repair pass, about $3.7 per day
At any step, one flag change rolls back: set the read flag gcp-redis-driver-search-index to off, or the write flag gcp-redis-driver-index to false. With both new flags off, every deploy changes nothing.
new-stagingqrGen2 and v1Gen2 included.qrGen2 and v1Gen2 included.cardsRead per search: p99 at most 500, maximum at most the region's eligible drivers. At most 5 commands per search on a dense test set.GET /nearbyDrivers and one QR booking. The Driver search source log shows source: redis with the right user.service.masterZCARD drivers:geo, the status-change rate and Lua billing. The PR must show new load at or below 15.2K, or the rollout stops.updateLocation, redissynclocations, onTripCreateV2, onTripUpdateV2, engine_queue_onUpdateV2 (each shard), qrGen2, v1Gen2. Gate: the expected DD_VERSION on each function, the legacy flag unchanged, no {dsidx key in Redis.gcp-redis-driver-index true for Metro Manila. The reconcile log shows low drift and a steady heartbeat.shadow, except qrgen2 and v1gen2. Gate over at least 2,000 searches: 0 mismatches, cardsRead p99 at most 500, tail network out at most 5 MB/s.qrGen2 and v1Gen2Step P6v1Gen2 allows up to 1,000 instances at concurrency 1. Then shadow, then on from 1% to 100%.These questions are still open. Each must be answered before the step named after it.
updateTime? Unit test, then one preprod trigger.region_configs equal the driver's region value? Compare the values.drivers:geo?K_SERVICE value in each search function. From step S7, needed before step P3.qrGen2 and v1Gen2 search rate and peak instances. Before step P6.Acceptance: with the index on, riders get the same drivers as today.
w = 0 and bad positions leave the index correct. A storm test checks that no eligible driver goes missing.reconcilePositions moves a driver only when the hash is newer, and interval 0 turns it off. The stream loop moves AWS drivers, skips other entries, keeps the newest entry per driver, and a stalled loop stops the heartbeat.GCP_REDIS_URL, the index flag is never read.service = K_SERVICE.GET /nearbyDrivers and one QR booking with the index on.release/gcp-redis-index, cut from master, with a PR to master.feature/gcp-redis-index, with PRs to the release branch.feature/gcp-redis-index-staging, cut from new-staging, cherry-picking the same commits.driverSearch.util.js change, the re-slice and the sync test go to master only, because new-staging has no Gen 1 driverSearch.util.js.Only the locations collection stops being written and read. Trips, region_configs and other config stay in Firestore. Three phases after step P6 remove the three ways search uses locations. Each phase has its own PR, gate and rollback.
setDriverState(uid, patch), writes driver:{uid}. Every state writer calls it after its Firestore write. The trigger still feeds the index.Driver state not direct-written when it sees a change the helper did not write.redissynclocations skips both.gcp-redis-driver-state-source switches a region between trigger and direct. Rollback: set it back to trigger.drivers:geo every 300 s.TIME or a version stamped by the writer.index_stale no longer forces Firestore, and the geofirestore query builders are deleted..near( query on locations is left in the Gen 2 functions.| Reason | Result |
|---|---|
| Index answered | index |
index_stale | index, logged as stale |
redis_error, no_candidates, all_filtered | empty |
A Datadog monitor watches heartbeat age. A stale index is better than an empty result, because by then the index is the only driver data left.
drivers:geoThat is the path that caused the incident: about 349 HGETALL calls per search. A region-wide fallback to it would repeat the incident.
The AWS writer already writes GCP Redis, and reconcilePositions reads driver:{uid}. Phase F needs no AWS change.