Scalability

Prerequisites

• System design basics (see Related Topics)

What you'll understand

• What scalability means
• The difference between scaling up and scaling out
• Why stateless services scale so much more easily

Explanation

Scalability is a system’s ability to handle increasing load, more users, more requests, more data, by adding resources. A system scales well if you can keep up with growth by adding capacity, and scales poorly if growth forces a painful redesign. Scalability is about how gracefully a system grows, not how fast it is at low load (that is performance, see Related Topics).

There are two ways to add capacity. Vertical scaling (scaling up) means a bigger machine, more CPU and memory; it is simple but has a hard ceiling and gets expensive fast. Horizontal scaling (scaling out) means more machines working together behind a load balancer; it has effectively no ceiling and is how large systems grow, but it requires the system to be designed for it.

The key to scaling out is statelessness. If each request can be handled by any instance because instances keep no per-user state in memory, you can add instances freely and a load balancer can spread traffic across them (session data lives in a shared store instead). Once you scale out, the bottleneck usually moves to whatever is shared, most often the database, which is why caching, read replicas, and careful data design matter so much at scale. Find the real bottleneck before scaling; adding servers in front of an overloaded database helps nothing.

Examples

The two directions of scaling trade simplicity against headroom:

Vertical (scale up)  : one bigger server        - simple, hard ceiling
Horizontal (scale out): many servers + balancer  - needs stateless design, scales far

Common mistakes

• Keeping per-user state in server memory, so instances cannot be added freely.
• Scaling out the app while the database remains the real bottleneck.
• Reaching for scale before measuring where the load actually is.
• Assuming a bigger machine can solve every capacity problem.

Best practices

• Design services to be stateless so they scale out horizontally.
• Keep shared state in a database or cache, not in instance memory.
• Find the real bottleneck before adding capacity.
• Relieve the database with caching and read replicas as load grows (see Related Topics).

Further reading

• Microsoft, Design to scale out — https://learn.microsoft.com/en-us/azure/architecture/guide/design-principles/scale-out
• Donne Martin, System Design Primer: Scalability — https://github.com/donnemartin/system-design-primer#scalability