Global Scope / Shared Infrastructure Blast Radius

The Day One Engineer Broke the Entire Warehouse

It was a normal Tuesday. A developer needed to clean up some stale rows from the warehouse_zones table in the warehouse management system. They opened a database client, wrote a quick DELETE query to clean up the specific records they wanted, and executed it.

They had forgotten the WHERE clause.

In under a second, every single row in warehouse_zones was gone. Not a few test records. Not one zone. All of them. The entire packer workflow — every warehouse in every city — came to a halt. Pickers couldn’t receive tasks. Zones couldn’t be assigned. Orders piled up in queues with no one to pick them.

This is what engineers mean when they talk about blast radius. And blast radius is almost always larger than you think — because in production, almost everything is shared.

What Is “Global Scope / Shared Infrastructure”?

In a simple toy application, changing something only affects that thing. In a real production system at scale, almost nothing is truly isolated:

Your database is shared across dozens of services and hundreds of concurrent queries
Your Redis cluster stores everything from session data to cache to rate limits
Your Celery or Kafka workers pull tasks from shared queues that multiple feature teams push to
Your global config files control behaviour for every feature that reads them
Your DNS resolution is shared across every pod in the cluster
Your connection pool is a finite resource consumed by every request hitting your service

When you change something in shared infrastructure — even a “small” change — you’re not changing one thing. You’re potentially changing the behaviour for every consumer of that resource simultaneously.

This post is a forensic walkthrough of four types of shared infrastructure failures, with real incident patterns, the exact code patterns that caused them, and the safeguards that would have contained the blast radius.

Part of a bigger list: this is a deep dive into one entry from 10 Production Failure Patterns I Keep Seeing — here we zoom all the way in on shared-infrastructure blast radius.

Type 1: The Shared Database — One Bad Query, Whole System Down

A database is the ultimate shared resource. Every service writes to it, reads from it, and competes for the same connection pool, the same CPU, the same buffer cache. One badly-written query can starve every other query in the system.

Incident: Weekly Payout Bulk Insert — DB CPU 97%

A rider management system runs a weekly payout job. Every week, it bulk-inserts one row per active rider on the platform — a batch large enough to stress the database in ways a small test dataset never would. The job had been running fine for months.

Then one week, something changed invisibly in the background: a large data migration had altered the table statistics that the query planner uses to pick indexes. The planner, now working with stale statistics, chose the wrong index for the bulk insert queries. The result:

Scenario	Query time
Correct index	0.08 ms
Wrong index (stale stats)	1,710 ms

That’s a 21,000× slowdown. Multiply that across every rider record in the batch, and the database CPU hit 97% and stayed there. Every other query sharing that database — rider logins, order assignments, status updates — slowed to a crawl. A significant number of orders were lost during the incident window because riders couldn’t log in to accept them.

The code triggering this was straightforward:

# A simple-looking bulk insert — nothing obviously wrong
PayoutRecord.objects.bulk_create(
    payout_records,  # one row per active rider — a large batch
    update_conflicts=True,
    update_fields=['amount', 'status', 'updated_at']
)

The problem wasn’t the code. It was the assumption that the database would always execute this the same way. Query plans aren’t guaranteed — they’re chosen dynamically based on statistics, and those statistics go stale.

The fix: Run ANALYZE before any large bulk operation that depends on accurate statistics, and add monitoring that alerts when a critical table’s last-analyzed timestamp is too old.

-- Run before bulk operations on large tables
ANALYZE payout_records;

Incident: Order Database CPU 100% — Workflow-Driven Joins

An urgent feature needed to ship in a single day. A developer wrote a three-table join query and put it inside a Temporal workflow activity — the part of the code that runs the actual business logic. What they didn’t account for: Temporal workflows can run hundreds of activities in parallel on a shared worker pool.

// This query lives inside a Temporal activity
func (a *Activity) GetOrderDetails(ctx context.Context, orderID string) (*OrderDetails, error) {
    // A 3-table join — looks fine in isolation
    return db.QueryRow(`
        SELECT o.*, u.name, p.amount
        FROM orders o
        JOIN users u ON u.id = o.user_id
        JOIN payments p ON p.order_id = o.id
        WHERE o.id = $1
    `, orderID)
}

When the feature launched and the workflow engine started running hundreds of instances in parallel, this query fired hundreds of times concurrently. Each one needed a DB connection. The connection pool saturated. The DB CPU hit 100%. Every other service sharing that database slowed down.

The fix: Any query that will be executed at high concurrency (inside a workflow activity, a goroutine pool, a parallel batch job) must be reviewed for cost at concurrency, not just cost for a single execution. Add execution plans to your PR review for any query inside a parallelised path.

Type 2: The Shared Redis Cluster — One Mistake, All Users Affected

Redis is a single-threaded in-memory store. It is phenomenally fast under normal conditions, but it has a hard ceiling: when memory fills up, it either starts evicting data silently or crashes entirely. Because Redis is typically shared across features, one team’s misconfiguration becomes everyone’s problem.

Incident: MONITOR Left Running on Prod

An engineer was debugging a production issue and opened a Redis MONITOR connection to see commands in real time. They fixed the issue and moved on — but forgot to close the MONITOR session.

MONITOR works by streaming a copy of every single Redis command to the subscriber. At production traffic volume, this meant Redis was generating thousands of command copies per second and buffering them into an open client connection. Memory usage began climbing. Over three hours, it climbed to 100%.

# This was left running and never closed
redis-cli MONITOR
# Streams every command: SET, GET, EXPIRE, everything
# At high production request volume, this buffers a large number of lines/second into memory

The fix was to close the connection — but by then, the cluster had been under stress for hours.

The rule: MONITOR and DEBUG commands must be explicitly blocked on all production Redis clusters. There is no production debugging scenario that requires leaving MONITOR open for longer than the diagnostic read.

Incident: Homepage Cache Poisoning — A Silent Type-Confusion Bug

A 1-second scheduling gap between two content refresh jobs caused an empty dictionary {} to be written into the homepage cache under a UI-icon key. The code that read this cache was:

new_bottom_nav_icon = cache.get('bottom_nav_icon')

# BUG: this only catches None — not an empty dict
if new_bottom_nav_icon is None:
    new_bottom_nav_icon = fetch_from_db()

render_homepage(bottom_nav=new_bottom_nav_icon)  # Renders with {}

Because an empty dict {} is not None in Python, the cache miss was never detected. Every single user who loaded the homepage got a broken rendering. The cache had a TTL, so every request for the TTL duration served the broken version. A substantial number of orders were lost before the issue was found and the cache key was manually cleared.

This is a type confusion bug with global blast radius: one poisoned cache key affecting every single user simultaneously.

The fix: Always validate cache contents, not just cache presence:

new_bottom_nav_icon = cache.get('bottom_nav_icon')

# Correct: check for both None AND empty
if not new_bottom_nav_icon:
    new_bottom_nav_icon = fetch_from_db()
    cache.set('bottom_nav_icon', new_bottom_nav_icon, ttl=300)

Type 3: The Global Config File — One Edit, All Features Affected

A “global config” is any single file or data store that controls the behaviour of many independent features at once. Global configs are convenient — one place to change things. They are also among the most dangerous surfaces in a production system, precisely because their blast radius is proportional to how many features depend on them.

Incident: Return-Rules Global JSON — Nationwide Return Processing Down

The warehouse management system used a single large JSON config file to define business rules for return order processing. A developer needed to add a new rule. They opened the config, added their rule, and deployed.

What they didn’t know: there were two different code paths that loaded this config — a standard path that had been tested and a legacy path that hadn’t been touched in months. The legacy path had a different internal data model. When it tried to parse the new rule, it failed.

The result: two return order types stopped working across every warehouse in the country simultaneously. QA had tested the standard path and it worked fine — they didn’t even know the legacy path existed.

// Standard path — tested, worked fine
func ProcessReturnOrder(ctx context.Context, order Order) error {
    rules := configLoader.LoadRules()       // Initializes cache properly
    return applyRules(rules, order)
}

// Legacy path — untested, different data model
func ProcessReturnOrderLegacy(ctx context.Context, order Order) error {
    rules := configLoader.LoadRules()       // MISSING: cache init that standard path had
    return legacyApplyRules(rules, order)   // Parsed the new rule differently, panicked
}

The rule for global configs: Every edit to a shared config file must include a full audit of every code path that reads it. A feature flag should gate any structural change to a shared config, so rollback is a single toggle rather than a redeploy.

Incident: Shared Task Queue Deployment — All Workers Broken

The Celery task queue is a shared worker pool — all features push tasks to the same queue, and the same worker pods consume them. A developer deployed a new task schema that added a required field with no default value:

# New task definition — required field, no default
@app.task
def process_order(order_id: str, store_tier: str):  # store_tier is NEW, required
    ...

But Celery workers use async serialization — there may be thousands of tasks already serialized in the queue from before the deployment, still using the old format. When the new workers started consuming those old messages, every single one failed deserialization:

TypeError: process_order() missing 1 required argument: 'store_tier'

Because the queue is shared, old tasks from every feature that happened to be queued at that moment were affected. The deployment was rolled back, but not before a significant backlog of failed tasks accumulated.

The rule: Every new field in a shared task schema must have a default value or be handled gracefully by consumers of all versions:

# Safe: backward-compatible schema
@app.task
def process_order(order_id: str, store_tier: str = "standard"):  # Default protects old messages
    ...

Type 4: Shared DNS and Connection Pools — Silent Skew at Scale

Incident: Database Reader DNS Caching

Managed relational databases often provide a reader endpoint DNS name that resolves to one of potentially several read replicas. The expectation: connections will be distributed roughly evenly across replicas. The reality: DNS responses are cached by the OS and by connection pool libraries, and each pod resolves the DNS once at startup and keeps that cached IP.

The result is that pods that start around the same time all resolve to the same replica — because the DNS record hasn’t changed yet. In practice, one replica ends up carrying the overwhelming majority of read traffic, while others sit mostly idle.

# Connection pool — resolves DNS once at startup
db = create_engine(
    "postgresql://user:password@my-db-reader.cluster.region.rds.amazonaws.com/app",
    pool_size=20,
    # No DNS re-resolution on pool checkout — stale connections skew load
)

When a deployment triggers all pods to restart within seconds of each other, they all resolve the same IP, overload one replica, and that replica’s CPU spikes while others are underutilized.

The fix: Force DNS re-resolution on connection checkout, or use a connection pooler that sits in front of the database and handles routing:

# Force fresh DNS resolution instead of relying on cached IPs
db = create_engine(
    "postgresql://...",
    pool_size=20,
    pool_pre_ping=True,  # Validate connections before use
    connect_args={"connect_timeout": 5},
)
# Or: stagger pod startups with a podDisruptionBudget to avoid thundering-herd DNS hits

The Mental Model: Blast Radius Before You Touch Anything

Every time you’re about to change something in production, ask yourself: “What else reads or writes this resource?”

Here’s a simple decision tree to use before any change:

You're about to change something.
        │
        ▼
Is this resource shared? (DB table, Redis cluster, config file, queue, DNS)
        │
       YES
        │
        ▼
Who else reads/writes it?
List all services, workers, cron jobs, scripts.
        │
        ▼
What happens to each of them if your change breaks?
        │
        ▼
Can you add a feature flag / canary / migration path
to narrow blast radius to a subset?
        │
        ▼
Plan the rollback FIRST. Then deploy.

Summary: The 5 Rules for Shared Infrastructure Safety

#	Rule	What it prevents
1	Always write WHERE clauses. Peer-review all DELETEs/UPDATEs	Whole-table data wipe
2	Run ANALYZE before bulk operations on shared tables	Query planner choosing wrong indexes under stale stats
3	Block MONITOR/DEBUG on all prod Redis clusters	Memory leak from forgotten debug sessions
4	Validate cache contents, not just cache presence (`if not val`, not `if val is None`)	Cache poisoning with empty/invalid values
5	All new fields in shared task/event schemas must have defaults	Breaking all workers on shared queues

The deeper problem is cultural, not technical. Every piece of shared infrastructure — a database, a Redis cluster, a config file, a Kafka topic — is a social contract between every team that uses it. When you change it, you’re not just changing your own feature. You’re potentially changing the experience for every engineer and every user who depends on it.

The engineers who caused these incidents weren’t careless or incompetent. They were working quickly under pressure, and they didn’t have a mental model that included the full scope of what they were touching. The fix isn’t to slow down. It’s to build habits — and tooling — that make the blast radius visible before the change happens, not after the pager fires.

The next time you’re about to DELETE FROM, modify a global config, or push a new task schema to a shared queue — pause for five seconds and ask: “Who else is in here with me?”