How Things Work

Scalability: Scale Up vs Scale Out

How systems grow to handle more load — the trade-offs between buying a bigger machine and running many smaller ones, and why statelessness decides which is even possible.

How It Works

Scalability is a system's ability to handle growing load by adding resources. There are two directions: scale UP (a more powerful machine) or scale OUT (more machines working together). Scaling up is simpler but capped; scaling out is unbounded but demands stateless, distributed design.

Vertical Scaling (Scale Up)

Move the workload to a bigger machine — more CPU, RAM, and faster disks. It is the simplest option (no code changes) and keeps everything on one box. But it has a hard ceiling: there is a largest instance you can buy, resizing usually means a reboot, and that one machine is a single point of failure.

Horizontal Scaling (Scale Out)

Run many identical instances behind a load balancer. Capacity grows almost linearly as you add nodes, there is no practical ceiling, and losing one node only removes a fraction of capacity. The cost: your application must be stateless and you now run a distributed system.

Statelessness Is the Enabler

Horizontal scaling only works if any instance can handle any request. That means no in-memory sessions, no local file uploads, no sticky data on the node. Push that state into shared services — Redis for sessions, object storage for files, a database for durable data.

The Bottleneck Moves

Adding web servers just pushes the pressure downstream. Soon the database, cache, or a third-party API becomes the limit. Real scaling is an iterative hunt: find the current bottleneck, relieve it (read replicas, caching, queues), then find the next one.

Key Concepts

♻️Stateless Service

An instance that stores no client-specific data between requests. Any replica can serve any request — the prerequisite for scaling out.

↕️Vertical Scaling

Increasing the resources (CPU/RAM) of a single machine. Simple but bounded by the largest available instance.

↔️Horizontal Scaling

Adding more machines and spreading load across them. Near-linear capacity growth with no hard ceiling.

📌Sticky Sessions

Pinning a user to one server so in-memory state survives. A crutch that undermines true statelessness and rebalancing.

📈Linear Scalability

The ideal where N instances handle ~N× the load. Shared resources and coordination overhead erode it in practice.

💥Single Point of Failure

A component whose failure takes down the whole system. One big vertical box is the classic example.

Stateless = scalable

tsx

1// The single rule that makes horizontal scaling possible:
2// keep your servers STATELESS.
3
4// ❌ Stateful — breaks when a second instance appears
5let sessions = {};                 // lives in THIS process's memory
6app.post("/login", (req, res) => {
7  sessions[req.user.id] = Date.now();   // lost if the next request
8});                                      // lands on a different server
9
10// ✅ Stateless — any instance can serve any request
11app.post("/login", (req, res) => {
12  // session state lives in a SHARED store, not in the process
13  await redis.set(`session:${req.user.id}`, token, "EX", 3600);
14});
15
16// Now you can run 1 or 100 copies behind a load balancer.
17// No instance "owns" a user — requests are interchangeable.

💡

Why This Matters

Almost every internet-scale system is horizontally scaled — it's the only way to grow past one machine and to survive node failures. Interviewers probe this constantly, and the design choices (statelessness, shared session stores, where the bottleneck moves) shape your entire architecture.

Common Pitfalls

⚠Assuming 'add servers' is free — coordination, shared databases, and chatty network calls erode linear scaling fast.

⚠Hidden in-memory state (sessions, caches, counters) that breaks the moment a second instance appears.

⚠Relying on sticky sessions instead of fixing statelessness — it limits rebalancing and concentrates load.

⚠Scaling the web tier while ignoring the database, which quickly becomes the real bottleneck.

⚠Over-provisioning 'just in case' instead of using auto-scaling tied to real metrics (CPU, queue depth, latency).

Real-World Use Cases

1Black Friday Traffic Spike

Scenario

An e-commerce site that comfortably serves 200 req/s suddenly faces 2,000 req/s when a sale goes live. Pages time out and the cart service falls over.

Problem

The team's only lever was a bigger database server, but they had already bought the largest instance the cloud offered. Vertical scaling was exhausted and they had no way to absorb the surge.

Solution

They made the web and cart tiers stateless (sessions moved to Redis), put them behind an auto-scaling group, and scaled horizontally from 4 to 40 instances during the event. The database was relieved with read replicas and a cache layer.

💡

Takeaway: Plan for horizontal scaling before you need it. Statelessness is cheap to design in early and painful to retrofit under load. Vertical scaling buys time; horizontal scaling buys headroom.

2The Hidden Stateful Server

Scenario

A team scales out their API to 6 instances behind a load balancer. Intermittently, users get logged out or see another user's draft.

Problem

One endpoint still cached per-user data in a module-level variable. With 6 instances, a user's requests bounced between processes that each held different in-memory state — classic non-deterministic bugs.

Solution

They audited for process-local state, moved sessions and drafts into Redis, and removed sticky-session config so the load balancer could rebalance freely. The bugs vanished.

💡

Takeaway: Scaling out turns hidden in-memory state into visible, intermittent bugs. If adding a second instance breaks something, you've found stateful code that must be externalized.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.