Your API Is Fast. So Why Is Your System Still Slow?
The simple difference between latency, throughput, p90, p99, and scalability that every developer should understand
Your API responds in 50 milliseconds.
Sounds great, right?
Now imagine 10,000 users hitting that API at the same time.
Suddenly, that “50ms API” starts taking 2 seconds. Then 5 seconds. Some requests start timing out.
So what happened?
Did the API suddenly become slower?
Maybe.
But there is another possibility: the API was always fast, but the system couldn't handle the amount of traffic coming toward it.
This is where two system design concepts become extremely important:
Latency and throughput.
They sound similar. They are often discussed together. But they answer two completely different questions.
Latency asks: How fast is one request? Throughput asks: How much work can the system handle?
Once you understand this difference, a lot of system design discussions start making much more sense.
Let's start with latency
Imagine you open an application and click:
“Show my orders.”
Your browser sends a request to the backend.
Client
|
| GET /orders
↓
Server
|
| Response
↓
Client
Suppose the entire operation takes 200 milliseconds.
That 200ms is the latency from the client's point of view.
In simple terms:
Latency is the time taken for a request to travel through the system and get a response.
But there is something important here.
The server doesn't just receive the request and immediately send a response.
A lot can happen in between.
What actually happens during an API call?
Consider this request:
GET /orders/123
The journey could look something like this:
User
|
↓
Internet
|
↓
Load Balancer
|
↓
API Server
|
├──→ Database
|
├──→ User Service
|
└──→ Payment Service
|
↓
API Server
|
↓
Load Balancer
|
↓
User
There are many places where time can be spent.
For example:
Network travel → 50ms
Request processing → 20ms
Database call → 80ms
Other service call → 40ms
Response travel → 60ms
--------------------------------
Total → 250ms
So when someone says:
“This API has 250ms latency.”
there may be much more happening underneath.
A useful way to think about it is:
Total Latency
=
Network Latency
+
Processing Latency
Network latency includes the time involved in moving data across the network.
Processing latency is everything the server needs to do after receiving the request — parsing data, calling databases, calling other services, running business logic, creating the response, and so on.
And this leads to an important debugging question:
If my API takes 500ms, where exactly are those 500ms being spent?
That is a much better question than simply saying:
“The API is slow.”
Average latency can hide a problem
Let's say your API receives 10 requests.
Their response times are:
100ms
110ms
120ms
105ms
115ms
100ms
125ms
110ms
120ms
2000ms
The average might still look reasonable.
But one user waited 2 seconds.
Now imagine you have millions of requests.
A small percentage of slow requests can affect a very large number of users.
This is why system designers often look beyond average latency.
Two particularly useful numbers are:
p90 and p99.
What does p90 mean?
Imagine you collect the latency of 100 requests and sort them from fastest to slowest.
If your:
p90 = 500ms
it means roughly 90% of requests completed within 500ms.
The remaining 10% took longer.
Similarly:
p99 = 2 seconds
means roughly 99% of requests completed within 2 seconds, while the slowest ~1% took longer.
This is useful because the slow tail of your system can tell you things that an average completely hides.
Think about an e-commerce website.
If 99% of users get their product page in 200ms but 1% wait 5 seconds, the average alone doesn't tell the whole story.
So instead of asking only:
“What's the average latency?”
you should also ask:
“What are my p90 and p99 latencies?”
These percentile measurements are also useful when setting monitoring and alerting thresholds.
Now let's talk about throughput
Latency was about:
How fast?
Throughput is about:
How much?
Suppose one server can process:
100 requests/second
Its throughput is:
100 requests/sec
If we have five identical servers:
Load Balancer
/ | | \
↓ ↓ ↓ ↓
Server Server Server Server
100 100 100 100
Ignoring bottlenecks for the moment, we could theoretically handle around:
5 × 100
= 500 requests/sec
That's throughput.
In application systems, you'll often hear TPS, which means:
Transactions Per Second
It's essentially a way of describing how many operations or requests a system can process in a given period.
A simple analogy: latency is the speed of a car, throughput is the road
Here's an analogy I find useful.
Imagine a road.
A car takes:
10 minutes
to travel from point A to point B.
That's similar to latency.
Now ask:
How many cars can this road handle every minute?
That's similar to throughput.
You could have a very fast road where each car reaches its destination quickly, but if there is only one lane, the number of cars you can handle may still be limited.
Now build a wider road.
Before:
🚗
🚗
🚗
After:
🚗 🚗 🚗 🚗 🚗
🚗 🚗 🚗 🚗 🚗
More cars can pass through at the same time.
That's an increase in throughput.
The same idea appears in distributed systems.
How do we increase throughput?
One obvious answer is:
Add more servers.
Suppose one server can handle:
100 requests/sec
You start with:
1 server
↓
100 requests/sec
Then traffic grows.
So you add another:
2 servers
↓
~200 requests/sec
Then another:
5 servers
↓
~500 requests/sec
A load balancer can distribute incoming requests across those servers.
Users
|
↓
Load Balancer
/ | \
↓ ↓ ↓
Server Server Server
This is horizontal scaling.
Instead of making one machine endlessly more powerful, you add more machines.
But there's a catch.
You cannot just keep adding servers forever.
Why?
Because servers cost money.
And more importantly, your server may not even be the bottleneck.
Before adding servers, check concurrency
Imagine you have a server that is only using:
CPU = 20%
but you're already thinking:
“Let's add ten more servers.”
Maybe you should first ask:
Can this server handle more concurrent requests?
A server might be capable of handling multiple requests at the same time.
Instead of:
Server
|
└── Request
you might have:
Server
├── Request
├── Request
├── Request
├── Request
└── Request
If the application is designed properly, increasing concurrency can allow existing infrastructure to handle more work.
For example, a simplified scenario might look like:
Before:
5 servers
× 1 concurrent request
= 5 active requests
After:
5 servers
× 15 concurrent requests
= 75 active requests
The exact numbers aren't important.
The important idea is:
Throughput depends not only on the number of servers, but also on how much concurrent work those servers can safely handle.
But don't blindly increase concurrency
This is where things get interesting.
More concurrency sounds great.
Until your CPU reaches its limit.
Or memory becomes the bottleneck.
Or your database connection pool gets exhausted.
Or another service cannot handle the additional traffic.
For example:
Users
|
↓
API Servers
|
↓
Database
Maybe your API servers can handle 50,000 requests/sec.
But your database can only handle 10,000.
Your API isn't the real bottleneck anymore.
The database is.
This is one of the most important lessons in system design:
Improving one component doesn't necessarily improve the entire system.
You need to find the bottleneck.
Latency and throughput can move independently
This is where many people get confused.
Consider two systems.
SystemLatencyThroughputA20ms100 req/secB200ms10,000 req/sec
Which one is better?
There isn't enough information to answer that.
If I'm building a highly interactive application, I may care heavily about latency.
If I'm processing millions of background jobs, throughput may be more important.
The requirements decide which metric matters more.
And sometimes you need both.
For example, imagine a payment API.
You don't want:
Low latency
+
Very low throughput
or:
High throughput
+
Terrible latency
You ideally want a system that can process a large number of requests while keeping response times within an acceptable range.
That's the real engineering challenge.
Here's a practical example
Imagine you're building a food delivery application.
A customer opens the app and requests nearby restaurants.
Your API does this:
GET /restaurants?lat=...&lng=...
The request goes through:
Client
↓
Load Balancer
↓
Restaurant API
↓
Database
↓
Restaurant API
↓
Client
Initially:
Average latency = 100ms
p99 latency = 300ms
Throughput = 1,000 req/sec
Everything looks good.
Then your application becomes popular.
Traffic increases to:
5,000 req/sec
Suddenly:
Average latency = 400ms
p99 latency = 3 seconds
What should you do?
A beginner might immediately say:
“Add more servers.”
Maybe that's correct.
But first I'd investigate.
High latency
|
↓
Where is time spent?
/ \
/ \
Network Processing
|
┌───────┼────────┐
↓ ↓ ↓
CPU Database API call
Maybe the database is overloaded.
Maybe the API servers have enough CPU but are waiting for database connections.
Maybe a downstream service is slow.
Maybe the network path has become expensive.
Maybe a cache could remove repeated database queries.
This is why understanding latency is more useful than simply looking at one number.
And this is where monitoring becomes important
A production system shouldn't just tell you:
API is healthy
You want to know things like:
Requests/sec
Average latency
p90 latency
p99 latency
Error rate
CPU usage
Memory usage
Database latency
Database connections
Now you can start asking useful questions.
For example:
Traffic increased by 2×. Did latency increase?
CPU is only 30%. Why is p99 suddenly 4 seconds?
Throughput is increasing, but database latency is also increasing. Is the database our bottleneck?
Average latency looks normal. Why did p99 suddenly jump?
Those are the kinds of questions that lead to real system-design decisions.
The mental model I use
When I hear latency, I think:
LATENCY
"HOW FAST?"
Client ───────────────→ Server
time taken
When I hear throughput, I think:
THROUGHPUT
"HOW MUCH?"
requests
↓
┌───────────┐
│ SYSTEM │
└───────────┘
↓
requests/sec
And when someone says:
“We need to scale.”
I immediately want to know:
Scale what?
Latency?
Throughput?
Both?
And what's currently limiting us?
One final distinction worth remembering
Here's a small cheat sheet:
Latency
→ How long does one request take?
Throughput
→ How many requests can we process per second?
p90
→ 90% of requests are faster than this value.
p99
→ 99% of requests are faster than this value.
TPS
→ Transactions per second.
Horizontal scaling
→ Add more servers.
Concurrency
→ Handle more work at the same time.
If you remember only two things, remember these:
Latency = How fast Throughput = How much
Everything else builds on top of that.
Final thought
One thing I have learned from system design is that “fast” doesn't automatically mean “scalable.”
A server can respond to a request incredibly quickly and still fall apart when thousands of requests arrive together.
And the opposite can also happen. A system can process a huge amount of traffic while individual requests take longer than we'd like.
That's why performance isn't represented by one magic number.
You need to understand the relationship between latency, throughput, concurrency, resource utilization, and bottlenecks.
So the next time someone tells you:
“Our API has 50ms latency.”
don't stop there.
Ask:
“Great. How many requests per second can it handle?”
And then ask the question that matters even more:
“What happens when traffic becomes 10× larger?”
That's when you're no longer just thinking about an API.
You're thinking about a system.