Read Replicas, Caching, And The Myth Of Infinite Scalability

Scaling Reads Sounds Easier Than It Is

At some point, every growing backend has the same conversation.

"The database is getting hot. Let's add read replicas."

Or:

"This endpoint is slow. Let's cache it."

Both ideas can be correct. Both can also create new bugs that are harder to explain than the original slow query.

Read replicas and caches are not magic performance buttons. They are trade-offs. They help you scale reads by accepting complexity around freshness, consistency, invalidation, and failure behavior.

It's like adding more checkout lanes at a store. Throughput improves, but now you need staff, coordination, line balancing, and a plan for when one lane's scanner breaks.

Read Replicas Help With Read Load

A read replica is a copy of your primary database used for read queries.

The primary handles writes:

App -> Primary DB -> writes

Replicas handle reads:

App -> Read Replica -> reads

This can reduce pressure on the primary database, especially for dashboards, reports, feed reads, and public browsing traffic.

But replicas usually receive changes asynchronously. That means they can lag behind the primary.

Replica Lag Creates Weird User Experiences

Imagine this flow:

1. User updates profile. The write goes to the primary.
2. App redirects to profile page. The read goes to a replica.
3. Replica is behind. The user sees the old profile data.

From the user's perspective, the save button lied.

The backend may be technically working, but the product feels broken.

This is the core rule: after a user writes data, be careful where you read from.

A practical solution is read-your-writes routing:

$user->update(['name' => $request->name]);

$user = User::on('mysql') // primary connection
    ->findOrFail($user->id);

For a short period after writes, read from the primary for that user/session/request.

Caching Makes Fast Things Possible And Wrong Things Faster

A cache stores data somewhere faster than the source of truth.

Example:

$product = Cache::remember("product:{$id}", 600, function () use ($id) {
    return Product::findOrFail($id);
});

This can protect your database and make hot endpoints much faster.

But now you have a second version of the truth.

If the product price changes, what happens to product:{$id}?

If the answer is "it expires in 10 minutes," then your system may show an old price for up to 10 minutes. That might be fine for blog posts. It might be unacceptable for checkout.

Cache Invalidation Is Product Logic

Cache invalidation is not just a technical chore. It is a product decision.

For each cached value, ask:

1. How stale can this be? One second, one minute, one hour?
2. Who sees the stale value? Admins, customers, internal systems?
3. What happens if it is wrong? Annoyance, lost money, security issue?
4. Can we invalidate precisely? One product key or a whole category page?

A safer write path may look like this:

DB::transaction(function () use ($product, $data) {
    $product->update($data);
});

Cache::forget("product:{$product->id}");

But even this has edge cases. What if cache deletion fails? What if another request rebuilds the cache between update and invalidation?

For high-risk data, avoid caching the value directly or use shorter TTLs, versioned keys, or write-through patterns.

Not All Data Deserves The Same Freshness

This is where backend engineering becomes product engineering.

Different data has different freshness needs:

1. User permissions. Should be very fresh. Stale permissions can become a security bug.
2. Product inventory. Usually needs to be fresh near checkout.
3. Homepage content. Can often be stale for minutes.
4. Analytics dashboards. Can often be stale for minutes or hours.
5. Exchange rates or taxes. Depends on business and compliance rules.

Treating all data the same is how teams either overcomplicate everything or accidentally serve dangerous stale data.

The Cache Stampede Problem

A cache stampede happens when many requests try to rebuild the same expired cache at once.

$stats = Cache::remember('dashboard:stats', 300, function () {
    return expensiveStatsQuery();
});

If this key expires during peak traffic, many requests may run expensiveStatsQuery() at the same time.

Now the cache did not protect the database. It coordinated an attack.

Common defenses:

1. Lock during rebuild. Only one request recomputes.
2. Use stale-while-revalidate. Serve old data while refreshing in the background.
3. Jitter TTLs. Avoid many keys expiring at the same second.
4. Pre-warm hot keys. Refresh before users trigger expensive rebuilds.

Read Replicas Can Also Make Migrations Riskier

Large schema changes can create replication lag. If the primary applies a big DDL operation and replicas fall behind, your read layer may become stale or unhealthy.

That matters during deployments.

A migration is not finished just because it ran on the primary. You need to know what happened to replicas, app code, and query routing.

This is why "just add replicas" is not a scaling strategy by itself. It's an operational commitment.

Better Patterns For Real Systems

Use replicas and caches deliberately:

1. Route critical reads to primary after writes. Especially confirmation pages and profile updates.
2. Classify data by freshness. Security-sensitive data should not use casual caching.
3. Use TTLs as safety nets, not your only invalidation plan. Time-based expiry is simple but blunt.
4. Monitor replica lag. Do not discover lag from customer screenshots.
5. Measure cache hit rate. A cache with poor hit rate may add complexity without benefit.
6. Have a fallback. Decide what happens when Redis or a replica is down.

Final Tips

I like caching and replicas, but they're not free money. They're more like credit cards: useful, powerful, and dangerous if you pretend the bill never arrives.

Before adding a cache or replica, write down the consistency expectation in plain English. "Users must see their own profile changes immediately" is more useful than "cache this for performance."

Scale carefully. Fast and wrong is still wrong 👊