17 Nov 2026 · Pacific time · spend: 0 US dollars
Multi-region design, per app class
MEASURED we ran it · FROM DOCS vendor page · MY ESTIMATE a guess, stated · NOT TESTED
The 6 answers
1YugabyteDB "write anywhere, copy later"?
Yes, it exists. It silently loses writes.
20 of 20 same-moment updates were lost. No error. MEASURED
Its own docs: "not recommended for YSQL". Keep our design.
Details2Can quests, groups, friends be class 2?
Yes. All of them. Drop class 3.
Every row has one home. Only small pieces cross.
Left for everyone: public search cards, unique names.
Details3Will copies cost a ton in transfer?
No. About 5 US dollars a month at 1 million users.
Copy all, 2 landscapes, normal use. Likely 0 inside free allowances.
The real cost is disk: about 200 a month after a year, growing.
Details + calculator4Cache invalidation?
Read copies need "wait, then tombstone", not just a delete.
Plain delete: 83 keys stuck. Wait-then-tombstone: 0. MEASURED
Class 2 keeps the decided delete-on-write.
Details5Read-after-write?
Class 2: free. Class 1: a commit token.
No token: 500 of 500 missed their own write. Token: 0 of 500. MEASURED
The library does it. Developers write nothing.
Details6One card per class
Class 1 one writer. Class 2 home region (default).
Plus one switch inside class 2: "copy all public rows".
See the cards1 · YugabyteDB: "multi-master, eventual consistency"
What it is FROM DOCS
- Two separate clusters, one per region. Both take writes.
- Changes copy to the other side later.
- Conflict rule: "last-writer-wins". The later timestamp overwrites the other.
- "This mode is not recommended for YSQL deployments."
- "Unique constraints cannot be guaranteed."
- Triggers fire on one side only. Sequences and schema changes are not copied.
- The safe (transactional) copy mode "does not support bidirectional replication".
Sources: xCluster docs, global app patterns (checked 17 Nov 2026).
What we measured MEASURED
Our Yugabyte spike, 17 Nov 2026. Two regions, 180 ms apart.
| Test | Result |
|---|---|
| Copy delay, one way | 181 ms |
| Singapore +10 and US +20 on one row, at once | 0 of 20 rows correct |
| Balance 100; pay 30 and pay 50 at once | both "succeed"; both show 70 |
| Same new name claimed on both sides | both accepted; one claim vanished |
| Errors the app saw | none |
Why writes were slow in the first place
- That test kept every row safe from a region loss.
- A safe write must reach a second region: 256 ms in Singapore. MEASURED
- Yugabyte can keep a row in one region only: 0.8 ms. MEASURED But the row is gone while that region is down.
- So you pick per table: fast and not region-safe, or safe and slow. That is physics.
When xCluster two-way would fit MY ESTIMATE
- Only if every row has exactly one writing region.
- That is our home-region rule. Postgres copies (variant B2) already give it, on our current vendors.
| Point | Yugabyte two-way | Our Postgres copies (B2) |
|---|---|---|
| Local write | fast, in region | about 3 ms MEASURED |
| Same row from 2 regions | last write wins, silent | refused; owner decides |
| Unique names | not guaranteed | 20 of 20 races, one winner |
| What we run | a new database system | PlanetScale / Neon, already chosen |
2 · Re-cut the classes, feature by feature
One rule for all features: each row has one owner landscape. Only the owner writes it.
Others get a copy, fetch it, or send a command to the owner.
Who owns what
| Feature | Who owns the row | What crosses regions |
|---|---|---|
| Friends | each person owns their half of the edge | the edge, to the friend's landscape only |
| Private group | the group's home landscape | posts and members, only where members live |
| Public group | the group's home landscape | a small card (name, count, last active), to all |
| Friend quest | the creator's landscape | each person's "finished" fact |
| Feed | each post's author's home | friends' posts, fetched per page |
| Comments | the commenter's home | to the post's home only |
| Likes | the liker's home | per-landscape totals |
| Public search | each card's owner | cards, to all landscapes |
| Unique handle | one directory, one region | one call at claim time |
| Direct messages | the sender's home | each message, to the receiver's home |
Which class, and how much crosses
| Feature | Class | How much crosses |
|---|---|---|
| Friends | 2 | a few rows per friendship |
| Private group | 2 | 0 bytes if all members are local MEASURED |
| Public group | 2 + card | card updates, batched every 250 ms MEASURED |
| Friend quest | 2 | about 7 messages, 3 KB per quest MEASURED |
| Feed | 2 | one fetch per page, about 184 ms MEASURED |
| Comments | 2 | one row per comment |
| Likes | 2 | 44 messages for 4,950 clicks MEASURED |
| Public search | 2 + card | one card per person or group |
| Unique handle | directory | one call per claim, about 190 ms MEASURED |
| Direct messages | 2 | one row per message |
What is left for "every region writes the same thing"?
- Nothing in this list. "Every region writes" just means each region writes its own rows.
- Hard cases (seats, stock, first to claim, invite approval) are a command to the owner.
- Exactly once, about 190–270 ms. 60 joiners for 50 seats gave exactly 50. MEASURED
- Many regions editing one row at once (live shared documents) would need a merge type (CRDT). No product needs it today. NOT TESTED
Final class list MY ESTIMATE
| Class | For | In one line |
|---|---|---|
| 1 One writer | read-mostly apps | one primary, read copies everywhere |
| 2 Home region | everything else (the default) | your data at home, plus a small sharing kit |
| 2 + "copy all public rows" | apps needing rich search or feeds over everything | a switch, not a class: B2 copies of all public tables |
Class 2 sharing kit (the developer declares; the library does it):
- Card: copied to every landscape by the database.
- Fetch: read a far thing through your own landscape. 184 ms cold, 0.1 ms cached. MEASURED
- Command: ask the owner for a one-answer decision. Exactly once. MEASURED
- Directory: unique names.
3 · What does data transfer really cost?
Prices, fetched 17 Nov 2026 FROM DOCS
| Item | US dollars per GB |
|---|---|
| AWS Singapore → other AWS region | 0.09 |
| AWS US East → other AWS region | 0.02 |
| AWS Singapore → internet (first 10 TB) | 0.12 |
| AWS US East → internet (first 10 TB) | 0.09 |
| PlanetScale egress, Asia | 0.096 |
| PlanetScale egress, US East | 0.06 |
| PlanetScale cross-region read copy, Singapore | 0.11 |
| Neon egress, after the free amount | 0.10 |
| Item | Free amount |
|---|---|
| AWS internet egress | 100 GB a month, whole account |
| PlanetScale production database | 100 GB a month |
| Neon paid project | 500 GB a month (database copies count) |
| Cloudflare R2 | egress always free |
| Tigris | egress always free |
| Upstash (pay as you go) | 200 GB a month, then 0.03 per GB |
Disk: Neon 0.35 US dollars per GB a month. FROM DOCS
The model (every assumption)
| Input | Value | Tag |
|---|---|---|
| Wire bytes per copied row, per peer | 360 | MEASURED |
| Disk bytes per copied row | 400 | MY ESTIMATE |
| Daily users ÷ monthly users | 0.3 | MY ESTIMATE |
| Far reads already in your region | 50 % | MY ESTIMATE |
| Far reads that miss the 30 s cache | 50 % | MY ESTIMATE |
| Bytes per fetch | 4,000 | MY ESTIMATE |
| Searches per daily user per day | 2 (8,000 bytes each) | MY ESTIMATE |
| Price per GB (worst list price) | 0.12 | FROM DOCS |
| Activity | Public rows written per user per month | Far reads per daily user per day |
|---|---|---|
| Light | 30 | 10 |
| Normal | 120 | 40 |
| Heavy | 1,200 | 200 |
- "Normal" per daily user per day: about 0.5 posts, 2 comments, 10 likes, a few edges.
- Copy all: every public row goes to every landscape.
- Small surface: only cards are copied; the rest is fetched.
- Copy nothing: everything is fetched; search asks every landscape.
Results, normal activity, US dollars per month (transfer)
2 landscapes
| Monthly users | Copy all | Small surface | Copy nothing |
|---|---|---|---|
| 10,000 | 0.05 | 0.22 | 0.39 |
| 100,000 | 0.52 | 2.17 | 3.89 |
| 1,000,000 | 5.18 | 21.73 | 38.88 |
3 landscapes
| Monthly users | Copy all | Small surface | Copy nothing |
|---|---|---|---|
| 10,000 | 0.10 | 0.29 | 0.63 |
| 100,000 | 1.04 | 2.91 | 6.34 |
| 1,000,000 | 10.37 | 29.06 | 63.36 |
Same in GB per month, 2 landscapes
| Monthly users | Copy all | Small surface | Copy nothing |
|---|---|---|---|
| 10,000 | 0.4 | 1.8 | 3.2 |
| 100,000 | 4.3 | 18.1 | 32.4 |
| 1,000,000 | 43.2 | 181.1 | 324.0 |
GB per month, 3 landscapes
| Monthly users | Copy all | Small surface | Copy nothing |
|---|---|---|---|
| 10,000 | 0.9 | 2.4 | 5.3 |
| 100,000 | 8.6 | 24.2 | 52.8 |
| 1,000,000 | 86.4 | 242.2 | 528.0 |
Heavy activity, 1 million users: copy all 52 (2 landscapes) or 104 (3); copy nothing 125 or 179 US dollars a month.
Extra disk for the copies, after one year (US dollars per month)
| Monthly users | Copy all, 2 landscapes | Copy all, 3 landscapes | Small surface, 3 landscapes |
|---|---|---|---|
| 10,000 | 2 | 4 | 0 |
| 100,000 | 20 | 40 | 0.03 |
| 1,000,000 | 202 | 403 | 0.28 |
Where the money really goes
- Disk for copies ("copy all"). It grows every month. At 1 million users after one year it is about 40 times the transfer cost.
- Fetching moves more bytes than copying. People read far more than they write. Still small.
- One-time copies. Adding a landscape, or a resync after a region loss, sends all public data once. 1 million users, one year: about 518 GB, about 62 US dollars. MY ESTIMATE
- Photos and video are not in this. They go through Tigris or R2: 0 egress. Never copy files between regions.
- Answers sent to phones cost the most, but every class pays that, single region too.
- Class 1 read copies stream the whole database log: about 1.6 KB per write (19 MB a minute at 195 writes a second MEASURED). PlanetScale charges 0.11 per GB for that in Singapore, with no free amount.
How to cap it
- Default to the small surface. Turn on "copy all" only when the app needs it.
- Copy a time window, for example the last 90 days of posts. Fetch older ones.
- Row filters and column lists keep private rows and columns home. MEASURED
- Alert on egress per database and on held-back log per copy link.
- Stay inside free amounts: 100 GB (PlanetScale), 500 GB (Neon) per database.
4 · Cache invalidation
Already decided (design corpus)
- Caches stay per region and short-lived.
- One Upstash cache per product per landscape, shared by all its clusters.
- Delete on write; versioned fill; single-flight; at most 16 database reads at once per pod; a breaker.
- Why one shared cache: separate caches per cluster gave 35 % stale reads. MEASURED 15 Nov.
- Public docs on the CDN: 60 s cache header, purge twice.
The trap with read copies (class 1)
Local proof MEASURED
17 Nov 2026. Postgres 17.6 primary + read copy that applies 300 ms late. Valkey 8 as region X's cache. Own containers, removed after.
Event to region X after 90 ms. 50 writes a second, 8 readers, 80 % of traffic on 20 hot keys. 20 s per strategy.
| Strategy | Reads stale over 1 s | Keys stuck after settling | Cache misses |
|---|---|---|---|
| TTL only (30 s) | 54,996 | 117 | 208 |
| Delete on write | 12,461 | 83 | 1,216 |
| Wait, then delete | 0 | 0 | 1,199 |
| Versioned tombstone | 0 | 0 | 17,837 |
| Wait, then tombstone (run 2) | 0 | 0 | 1,177 |
Each run: about 63,000–67,000 reads, about 900 writes. Run 2 repeated delete-on-write (12,228 stale, 77 stuck) and tombstone (0, 0, 17,805 misses).
- "Stale over 1 s": the read returned a version replaced more than 1 s earlier. The copy is only 300 ms behind, so the cache caused it.
- "Stuck after settling": old values still in the cache 1.8 s after writes stopped.
- About a third of all reads were older than the newest write but inside the 300 ms copy delay. That is the copy delay itself. Only reading the primary avoids it.
The strategies in plain words
| Strategy | What it does | Verdict |
|---|---|---|
| TTL only | wait for expiry | wrong for up to the TTL |
| Delete on write | delete when the event arrives | refills from the late copy |
| Wait, then delete | delete only once the local copy has the write | works; needs a waiter per region |
| Versioned tombstone at once | mark "needs version N"; older fills are refused | works; about 15 times more cache misses |
| Wait, then tombstone | once the local copy has the write, mark "needs version N" | works, normal load: pick this |
Class 1 rule (proposed)
- Every write sends (key, new version, database position). The library does it from the outbox.
- A small waiter in each region checks its own read copy's position. Measured wait: about 215 ms after the event, for a 300 ms copy delay. MEASURED
- Once the copy has the write, the cache gets a tombstone: "this key needs version N or newer".
- A fill is stored only if it is not older (the decided versioned-fill script). This blocks a slow fill that read the old row.
- Single-flight per key (decided) keeps a hot key from flooding the copy.
- TTL stays as the backstop. If the waiter cannot reach the copy for 5 s, it sets the tombstone anyway.
Who clears region X's cache when region Y wrote?
| Class | Who clears it |
|---|---|
| 1 | the write event (tombstone), sent to every region |
| 2, own data | nobody else: only the home caches it; delete on write |
| 2, fetched far data | 30 s TTL, plus the owner's change event (unfriend hid data 283 ms later MEASURED) |
| 2, copied cards | a database trigger on copy apply, or no cache at all (local read is 0.4–9 ms MEASURED) |
Per store
| Store | Class 1 | Class 2 |
|---|---|---|
| Cache (Upstash) | per region; tombstones from the write event | per landscape; own data |
| Key-value (Upstash) | one primary; fresh reads go to it | per landscape, never synced |
| Files (Tigris / R2) | new name per upload; nothing to clear | same |
| CDN (public docs) | 60 s header + purge twice (decided) | same |
Upstash Global read regions bill each copied write as a command: 100K writes with one read region = 0.4 US dollars. FROM DOCS
5 · Read-after-write
Database level
| Fix | How | Cost |
|---|---|---|
| Commit token (pick this) | keep the write's database position; read the copy once it has passed it | waits about the copy's delay |
| Route to primary for N seconds | after a write, this session reads the primary | far reads (about 180 ms) for N seconds |
| Synchronous apply on copies | every commit waits for far copies | every write pays the ocean again |
| Pin the session | keep a person on one database | does not help across regions |
Local proof MEASURED
A person in region X writes (through the primary), then reads their own row at once. Copy applies 300 ms late. 500 runs each.
| Read path | Own write missing (of 500) | Wait, middle | Wait, slowest 1 % |
|---|---|---|---|
| No token | 500 | n/a | n/a |
| Token, position checked in the same statement | 3 | 305 ms | 311 ms |
| Token, position checked first, then read | 0 | 304 ms | 311 ms |
0 fallbacks to the primary were needed (limit 1 s).
Per database
| Database | Read-after-write tool | Tag |
|---|---|---|
| Postgres 17 / 18 (PlanetScale, Neon) | compare the copy's replay position | MEASURED (17.6) |
| Postgres 19 | a built-in WAIT FOR LSN command | FROM DOCS (development docs) |
| PlanetScale | read copies only with the "replica" login suffix, not via its pooler | FROM DOCS |
| YugabyteDB | normal reads are always fresh; follower reads are old by a set bound | MEASURED |
| B2 copies (class 2 cards) | the copy's progress per source landscape vs the writer's position | NOT TESTED |
Cache level
| Fix | How |
|---|---|
| Write-through in the writer's region | after commit, put the new version in the local cache |
| Bypass window | the writer's session skips the cache for keys it just wrote |
| Version stamps | the session keeps "min version"; an older cached value counts as a miss |
Per class
| Class | Your own write, then read | Someone else's write |
|---|---|---|
| 1 | commit token + version check in the cache | after about the copy's delay |
| 2 | free: you read where you write | about 0.1 s plus distance (93 ms with 90 ms added MEASURED) |
| 2, command to a far owner | the answer goes into your local view at once | same |
What the developer writes: nothing. The library returns the token in a response header. The app sends it back. The read path checks it.
6 · One card per class
Class 1 · One writer
| Part | Design |
|---|---|
| Database | one primary; read copies in each region |
| Cache | per region; versioned tombstones from the write event |
| Key-value | one primary; fresh reads go there |
| Files | one bucket; new name per upload; served by the CDN |
| Migrations | once, on the primary |
| Read-after-write | commit token |
| Transfer cost | whole log to each copy region, about 1.6 KB per write |
| Developer writes | nothing extra |
| Tested | cache + token MEASURED locally; vendor copies NOT TESTED |
Class 2 · Home region (the default)
| Part | Design |
|---|---|
| Database | one per landscape; public cards copied (B2) |
| Cache | per landscape, own data; fetched data 30 s |
| Key-value | per landscape, never synced |
| Files | bucket per landscape; read anywhere via the CDN |
| Migrations | per landscape; copied tables: add first, remove later |
| Read-after-write | free for your own data |
| Transfer cost | 1 million users, normal: about 22 (2 landscapes) to 29 (3) US dollars a month |
| Developer writes | declares each table: private, card, fetch or command |
| Tested | copies, commands, fetch MEASURED locally; real vendors: separate check |
Class 2 with "copy all public rows"
| Part | Design |
|---|---|
| Database | every public row copied to every landscape (B2) |
| Transfer cost | 1 million users, normal: about 5 (2 landscapes) to 10 (3) US dollars a month |
| Disk cost | about 200 (2) to 400 (3) US dollars a month after one year, growing |
| Limit | about 5 landscapes (every pair needs its own link) |
| Why pick it | plain SQL search, joins and feeds over everything |
Not tested, vendor check, sources
Not tested
- Real distance and bandwidth. Every proof adds delay on one machine.
- The tombstone cache and commit token on PlanetScale or Neon read copies.
- Commit tokens across B2 copies.
- Fetch size (4,000 bytes is a guess).
- Activity levels are guesses, not app data.