What Really Happens When 1 Million Users Hit Your Website at the Same Time?

A beginner-friendly journey from one server to load balancers, microservices, queues, caching, databases, and CDNs.

What Really Happens When 1 Million Users Hit Your Website at the Same Time?

Imagine you built a website.

It works perfectly.

You have one server, one database, and a few hundred users.

Then one morning, something crazy happens.

Your website suddenly becomes popular. 100 users become 10,000. 10,000 become 100,000. And then…

1 million people open your website at almost the same time.

What happens now?

  • Does the server simply work harder?
  • Do you buy a bigger server?
  • What if that server crashes?
  • What if the database can’t handle the traffic?
  • What if users are coming from India, the US, Canada, and Europe?

And most importantly:

How do companies build systems that don’t fall apart when traffic suddenly explodes?

Let’s build the answer from scratch.

Start With One Server

Let’s forget about Amazon, Netflix, Google, or any large company for a moment.

Suppose you’re building a simple e-commerce application.

At the beginning, you might have something like this:

User
 |
 v
Server
 |
 v
Database

The server is just a machine running your application.

It could be a physical machine or a virtual machine in the cloud.

Your users send requests to it:

GET /products
GET /products/123
POST /orders

The server processes those requests and sends responses back.

Simple.

But there’s one problem.

A server has limited resources.

It has CPU, RAM, disk, network bandwidth, etc. As traffic increases, those resources start getting used up.

So let’s say your server looks like this:

CPU: 2 cores
RAM: 4 GB

Everything is fine until suddenly thousands of users start sending requests.

Eventually:

CPU → 100%
RAM → 100%
Requests → waiting
Response time → increasing

And eventually…

your application may crash.

You have probably seen something similar when a popular website becomes unavailable during exam results, ticket sales, or some major event.

So what do we do?

Option 1: Buy a Bigger Server

The first solution is pretty obvious.

If your server isn’t powerful enough, make it more powerful.

Maybe:

2 CPU + 4 GB RAM
 ↓
8 CPU + 32 GB RAM
 ↓
64 CPU + 128 GB RAM

This is called vertical scaling.

You are increasing the capacity of a single machine.

Think about it like a restaurant.

You have two cooks.

Suddenly more customers start arriving.

Instead of opening another kitchen, you give the same kitchen more resources:

“Let’s hire more cooks and make this kitchen bigger.”

Sounds good.

But there’s a problem.

A single machine still has a limit.

You can’t keep making one machine infinitely powerful.

And there’s another problem.

What if your traffic looks like this?

Traffic
 /\
 / \
 / \
_____/ \________

You need huge resources during the peak.

But what about the rest of the day?

You might be paying for a massive machine while using only a small part of it.

Cloud platforms can help by allowing resources to scale based on demand, but vertical scaling still has practical limits and can involve downtime depending on how resources are changed.

So we need a better idea.

Option 2: Add More Servers

Instead of making one server huge, why not use multiple servers?

 ┌── Server 1
 │
Users ───────┼── Server 2
 │
 └── Server 3

Now if one server can handle 1,000 requests, three servers can potentially handle much more traffic.

This is called horizontal scaling or scaling out.

And this is where things get interesting.

Because now we have a new question:

Which server should receive each request?

If the user doesn’t know which server is available, who decides?

Enter the Load Balancer.

The Load Balancer: The Traffic Police

A load balancer sits in front of your servers.

 ┌── Server 1
 │
Users → Load Balancer ├── Server 2
 │
 └── Server 3

Its job is simple:

Distribute incoming traffic across available servers.

One simple strategy is round robin:

Request 1 → Server 1
Request 2 → Server 2
Request 3 → Server 3
Request 4 → Server 1
Request 5 → Server 2
...

When traffic increases, you can spin up another server.

 ┌── Server 1
 ├── Server 2
Users → LB ──┼── Server 3
 └── Server 4

When traffic decreases, you can remove servers.

This is one of the major advantages of horizontal scaling.

A load balancer can also check whether a server is healthy before sending traffic to it.

So now we have:

Users
 |
 v
Load Balancer
 |
 ├── Server
 ├── Server
 ├── Server
 └── Server

Looks pretty good.

But there’s another problem.

What If You Have Multiple Services?

Imagine your e-commerce application grows.

You don’t want one giant application handling everything.

You might split it into different services:

Auth Service
Order Service
Payment Service
Product Service

And each service may have multiple servers.

For example:

 
 ┌── Auth Load Balancer
 │ └── Auth Server
 │
Users → ??? ├── Order Load Balancer
 │ ├── Order Server
 │ └── Order Server
 │
 └── Payment Load Balancer
 ├── Payment Server
 └── Payment Server

Now ask yourself:

If the request is /orders, where should it go?
If it’s /payment, where should it go?

We need something that understands the request and routes it to the correct service.

That’s where an API Gateway comes in.

API Gateway: The Reception Desk

Think about a hotel.

You walk into the reception.

You say:

“I need room 203.”

The receptionist doesn’t personally take you there.

They figure out where you need to go and route you appropriately.

An API Gateway works in a similar way.

 ┌── Auth Service
 │
User → API Gateway ├── Order Service
 │
 └── Payment Service

For example:

/api/auth/* → Auth Service
/api/orders/* → Order Service
/api/payments/* → Payment Service

The gateway becomes a centralized entry point for API requests and routes them to backend services.

It can also be involved in things like authentication and other common request-level policies.

Now our architecture is starting to look like a real production system.

But there’s another problem waiting for us.

Should Every Task Happen Immediately?

Suppose a customer successfully places an order.

You want to:

  1. Save the order.
  2. Send an email.
  3. Send an SMS.
  4. Send a WhatsApp notification.
  5. Update analytics.

Do we really want the order API to wait for all five things to finish?

Imagine your code doing something like:

await saveOrder();
await sendEmail();
await sendSMS();
await sendWhatsApp();
await updateAnalytics();
return "Order successful";

What happens if the email provider takes 3 seconds?

The user waits.

What if the email provider is temporarily down?

The order request might fail even though the order itself was successfully created.

This is where asynchronous processing becomes useful.

Enter the Queue

Instead of directly calling the email service, we can put a message into a queue.

Payment Service
 |
 v
 Queue
 |
 v
 Email Worker
 |
 v
 Email Provider

The payment service can basically say:

“Payment completed. Someone needs to send this email.”

And move on.

The worker processes the task in the background.

A simplified example:

// Payment service
await savePayment();
await emailQueue.push({
 userId,
 orderId
});
return "Payment successful";

The worker can then do:

while (true) {
 const job = await emailQueue.pop();
 await sendEmail(job);
}

Now the payment request doesn’t have to sit around waiting for an external email service.

And if traffic increases?

Add more workers.

 ┌── Worker 1
 │
Queue ────────┼── Worker 2
 │
 └── Worker 3

More workers → more parallel processing.

This is one reason queues are so useful in high-traffic systems.

But What If the Email Service Allows Only 10 Emails Per Second?

Here’s a real problem.

Your application might be capable of processing 10,000 jobs per second.

But an external service might say:

“You can send only 10 requests per second.”

Now you have a bottleneck.

A queue helps because the work can wait safely while workers process it at the allowed rate.

10,000 jobs
 ↓
 Queue
 ↓
10 jobs/sec
 ↓
Email Provider

The queue becomes a buffer between your system and the slower external system.

That’s much better than making your users wait.

One Event, Many Things

Now imagine a customer pays for an order.

You want multiple things to happen:

Payment completed
 |
 ├── Send Email
 ├── Send SMS
 ├── Send WhatsApp
 └── Update Analytics

This is where Pub/Sub and the fan-out pattern become useful.

Instead of the payment service calling every service individually, it can publish an event:

"PaymentCompleted"

Multiple services can listen for it.

 ┌── Email
 │
Payment Event────┼── SMS
 │
 ├── WhatsApp
 │
 └── Analytics

This is an example of an event-driven architecture.

One event can trigger work in multiple independent services.

But there is an important distinction.

A normal queue is useful when one consumer should process a job.

Pub/Sub is useful when multiple consumers need to react to the same event.

For example:

“A payment happened.”
  • Email cares.
  • SMS cares.
  • Analytics cares.

Fraud detection might care too.

That’s a perfect candidate for an event.

What If Something Fails?

Distributed systems fail.

It’s not a question of if.

It’s a question of when.

Suppose your worker receives a job:

Queue
 ↓
Worker
 ↓
Email Provider ❌

Maybe the email provider is temporarily unavailable.

Should we simply delete the message?

Probably not.

Instead, we can retry.

And if it keeps failing, we can move it to a Dead Letter Queue (DLQ) for later investigation or retry.

This is one of the big reasons queues are more than just “a list of tasks.”

They can help us build systems that tolerate temporary failures.

But Now Users Can Attack Our System

Let’s return to our 1 million users.

What if not all requests are legitimate?

Someone could send:

Request
Request
Request
Request
Request
Request
...

thousands of times per second.

Or a large number of machines could send requests simultaneously.

Now your infrastructure is doing unnecessary work.

We need rate limiting.

For example:

User → 5 requests/sec → Allowed
User → 6th request → Rejected

That’s why you may have seen errors such as:

429 Too Many Requests

Rate limiting controls how much traffic a client or system is allowed to generate within a period of time. Common approaches include token bucket and leaky bucket algorithms.

The important idea isn’t the algorithm yet.

It’s this:

Don’t let one client consume all the resources of your system.

Now We Have Another Problem: The Database

Look at what we’ve built.

Users
 ↓
API Gateway
 ↓
Load Balancers
 ↓
Microservices
 ↓
Database

Every service needs data.

And eventually, the database becomes the bottleneck.

Imagine 50 application servers all asking one database for data.

Server ─┐
Server ─┤
Server ─┤
Server ─┼──→ Database
Server ─┤
Server ─┘

The application layer can scale horizontally.

But if everything still depends on one database, we haven’t really solved the problem.

So what can we do?

Read Replicas

One common approach is to create database replicas for read-heavy workloads.

Conceptually:

 ┌── Read Replica
 │
Primary DB ───┼── Read Replica
 │
 └── Read Replica

Writes go to the primary.

Read-heavy workloads can be distributed to replicas.

For example:

INSERT order
 ↓
Primary DB

GET product
 ↓
Read Replica

This reduces the amount of read traffic hitting the primary database.

There is an important trade-off, though.

Replicas may not always have the newest data immediately.

Replication can have some delay.

So if you absolutely need the latest value, you may need to read from the primary instead.

That gives us another system-design question:

Do I need perfectly fresh data, or is slightly stale data acceptable?

That’s a question you’ll encounter again and again in distributed systems.

Why Query the Database Every Time?

Suppose millions of users keep asking:

GET /products/123

And the product information doesn’t change every second.

Why should we repeatedly hit the database?

We can keep frequently accessed data in memory.

That’s where a cache comes in.

For example, Redis is commonly used as an in-memory data store for caching.

The flow becomes:

User
 ↓
Service
 ↓
Cache
 ├── HIT → Return data
 │
 └── MISS
 ↓
 Database
 ↓
 Store in Cache
 ↓
 Return data

Pseudo-code:

let product = await cache.get("product:123");
if (!product) {
 product = await db.getProduct(123);
 await cache.set("product:123", product);
}
return product;

The next request can potentially be served from memory instead of hitting the database.

That means:

  • fewer database queries
  • faster responses
  • less pressure on the database

Caching is therefore not just about making things faster.

It can also help protect your database from unnecessary work.

But What About Users Around the World?

Now imagine your main infrastructure is located in the US.

You have users in:

India
Canada
US
Europe
Australia

Should every user travel all the way to your main server just to download an image?

Probably not.

Consider a product page on an e-commerce website.

The product image might be the same for thousands of users.

Why should every request travel to your origin server?

This is where a CDN — Content Delivery Network comes in.

CDN: Put Content Closer to Users

A CDN has servers distributed across different geographic locations.

Conceptually:

 CDN Edge
 / | \
 / | \
 India US Europe
 \ | /
 \ | /
 Origin
 Server

When a user requests a cacheable resource, such as an image, the CDN can serve it from an edge location closer to the user if it is already cached there.

For example:

User in India
 ↓
India CDN Edge
 ↓
Image found in cache
 ↓
Return immediately

No need to go all the way back to the origin server.

If the image isn’t cached:

User
 ↓
CDN Edge
 ↓
Cache MISS
 ↓
Origin Server
 ↓
CDN caches response
 ↓
User

The next user requesting the same resource may receive it directly from the CDN.

This reduces latency and also reduces traffic reaching your origin infrastructure.

And this is especially useful for static content such as:

  • Images
  • Videos
  • CSS
  • JavaScript
  • Other cacheable assets

So What Does Our 1-Million-User System Look Like Now?

We started with this:

User → Server → Database

And slowly discovered problems.

Too much traffic?

Horizontal scaling.

 ┌── Server
Users → LB ──┼── Server
 └── Server

Multiple services?

API Gateway.

Users
 ↓
API Gateway
 ├── Auth
 ├── Orders
 └── Payments

Slow background work?

Queue + Workers.

Service → Queue → Workers

One event, multiple consumers?

Pub/Sub / Fan-out.

 ┌── Email
Event ───────┼── SMS
 ├── WhatsApp
 └── Analytics

Too many requests?

  • Rate limiting.

Too many database reads?

  • Read replicas + caching.

Users distributed globally?

  • CDN.

Put everything together:

 ┌── Auth Service
 │
Users → CDN → API Gateway ── Order Service
 │
 └── Payment Service
 |
 v
 Queue
 |
 ┌──────┼──────┐
 ↓ ↓ ↓
 Email SMS WhatsApp



Services
 |
 v
 Cache
 |
 v
Primary Database
 |
 ├── Read Replica
 ├── Read Replica
 └── Read Replica

Suddenly, system design stops looking like a collection of random boxes.

Every box exists because we had a problem.

That’s the Part I Wish More Beginners Understood

When people start learning system design, they often memorize things like:

Load balancer → API Gateway → Redis → Kafka → CDN → Database

But that’s not really system design.

The better approach is to ask:

“What problem am I trying to solve?”

For example:

Problem: One server can’t handle the traffic.

→ Add more servers.

New problem: Who distributes traffic?

→ Load balancer.

New problem: We have multiple services.

→ API Gateway.

New problem: Some tasks don’t need to happen during the request.

→ Queue.

New problem: One event needs to trigger multiple services.

→ Pub/Sub / fan-out.

New problem: Too many requests can overload us.

→ Rate limiting.

New problem: Database reads are too expensive.

→ Cache / read replicas.

New problem: Users are far away from our servers.

→ CDN.

That’s a much better way to learn system design.

The Real Skill Isn’t Memorizing Components

Here’s the mental model I’d keep:

Traffic increases
 ↓
Find the bottleneck
 ↓
Ask why it's a bottleneck
 ↓
Introduce a solution
 ↓
That solution creates a new problem
 ↓
Solve the new problem
 ↓
Repeat

That’s basically how complex systems evolve.

And this is why system design interviews shouldn’t be treated as:

“Tell me what Redis does.”

A better question is:

“Why would you put Redis here?”

Because if you can’t explain the problem, knowing the name of the technology doesn’t help much.

One Last Question

Let’s say tomorrow your application suddenly goes viral.

You go from:

10,000 users
 ↓
1,000,000 users

Would your first reaction be:

“Let’s add more servers.”

Or would you first ask:

“Which part of my system is actually becoming the bottleneck?”

That difference is important.

Because a scalable system isn’t created by throwing more machines at every problem.

It’s created by finding bottlenecks, understanding why they exist, and designing the system around them.

Conclusion

We started with a simple application:

User → Server → Database

It was enough when the traffic was small.

But as users increased, new problems appeared.

The server became a bottleneck.

So we scaled horizontally.

Then we needed a load balancer.

Multiple services introduced routing problems, so we added an API Gateway.

Background tasks were slowing down requests, so we introduced queues and workers.

Multiple services needed to react to the same event, so Pub/Sub and fan-out patterns became useful.

Too much traffic required rate limiting.

The database became a bottleneck, so we introduced caching and read replicas.

And when users were spread across the world, a CDN helped bring content closer to them.

The interesting part is that none of these components should exist just because someone told you they’re “important system design components.”

They exist because a problem forced us to introduce them.

That’s the mindset I would take into a system design interview:

Don’t start by drawing boxes. Start by finding problems.

Because once you understand the problem, the architecture becomes much easier to design.

And honestly, that’s when system design starts becoming fun.

If you had to remove one component from this architecture, which one do you think would cause the biggest problem — and why?

From Tech By Neha Gupta

  • 👏 Enjoyed the article? Don’t forget to leave a clap.
  • 💬 Have thoughts or questions? Share them in the comments.