What a Bad Flash Sale Taught Me About Latency and Throughput

The night a flash sale taught me that average latency is a lie, and throughput isn’t what most engineers think it is.

It was 8:00 PM. Flash sale night. Our biggest one of the quarter.

Every dashboard I had open was green. CPU usage, fine. Memory, fine. API uptime, 100%. I remember actually leaning back in my chair thinking, “okay, this one’s going smoothly.”

Then the support channel exploded.

“Checkout button not working” “Stuck on payment page” “Order not placing, tried 5 times”

I pulled up the same dashboard everyone else was staring at. Still green. Still “healthy.” We got confused what’s wrong? Who was lying — the dashboard, or the customers?

Turns out, neither. I just didn’t understand the actual difference between two words I’d been using interchangeably for years: latency and throughput.

If you’ve ever had a system that looked “up” on paper while users were quietly rage-quitting your app, you already know this pain. Stick with me — by the end of this, you’ll never confuse these two again, and you’ll know exactly which one to blame the next time production goes sideways.

Latency: “How Fast,” Not “Is It Working”

Here’s the simplest way I explain latency to anyone new on my team, noobs included:

Latency is how long it takes for one request to make a round trip — from the moment your user clicks something, to the moment they see a result.

That’s it. Not whether it works. Not whether the server is “up.” Just — how long did they wait.

In our checkout flow, when a user hit “Place Order,” here’s roughly what happened behind the scenes:

Client (browser)
   |
   |  1. Request hits our API Gateway        →  ~15ms
   v
Checkout Service
   |
   |  2. Check inventory (DB call)           →  ~40ms
   |  3. Call payment gateway (external API) →  ~180ms
   |  4. Write order to database             →  ~35ms
   |  5. Send confirmation event to queue     →  ~10ms
   v
Response back to client                       →  ~15ms
-----------------------------------------------------
Total latency: ~295ms

Notice something? Only a small chunk of that time (the gateway hops, ~30ms) is actually “network latency.” The rest — inventory check, payment call, DB write — that’s processing latency. That’s the part you control. The network hop, you mostly can’t do much about.

This is the mistake I made early in my career: I used to blame “the network” for everything. Nine times out of ten, it wasn’t the network. It was my own code doing something slow, like calling the payment gateway synchronously when it didn’t need to block the rest of the response.

Here’s a dead-simple way to actually measure this in Node.js instead of guessing:

function timeIt(label, fn) {
  return async (...args) => {
    const start = performance.now();
    const result = await fn(...args);
    const duration = performance.now() - start;
    console.log(`${label} took ${duration.toFixed(2)}ms`);
    return result;
  };
}

// Usage
const checkInventory = timeIt('inventory-check', async (productId) => {
  return await db.query('SELECT stock FROM products WHERE id = ?', [productId]);
});
const chargePayment = timeIt('payment-gateway', async (order) => {
  return await paymentClient.charge(order);
});

Run this in production (sampled, not on every request — don’t be that engineer who logs everything and kills their own throughput), and suddenly you’re not guessing anymore. You know which step is slow.

Why “Average Latency” Almost Got Me Fired

On that flash sale night, our average latency was sitting at a comfortable 220ms. Totally fine, right?

Except averages hide the truth. Here’s the thing nobody tells junior engineers: if 9,000 users get a response in 100ms and 1,000 users get a response in 8 seconds, your average still looks pretty okay. But that’s 1,000 real people staring at a spinning wheel, ready to abandon their cart — or worse, tweet about it.

This is exactly why senior engineers talk about P90 and P99 instead of averages. Have you ever checked your P99 and been genuinely shocked by the number? I have. More than once.

P90 = 90% of your users had a latency below this number.
P99 = 99% of your users had a latency below this number. This is your worst-case-that-still-matters.

Here’s how you’d actually calculate that from a batch of response times, in plain JavaScript:

function getPercentile(latencies, percentile) {
  const sorted = [...latencies].sort((a, b) => a - b);
  const index = Math.ceil((percentile / 100) * sorted.length) - 1;
  return sorted[index];
}

const responseTimes = [95, 102, 110, 98, 4200, 105, 99, 8900, 101, 97]; 
// notice those two outliers hiding in there
console.log('P50 (median):', getPercentile(responseTimes, 50));
console.log('P90:', getPercentile(responseTimes, 90));
console.log('P99:', getPercentile(responseTimes, 99));

When I ran our actual production numbers that night, average was 220ms — but P99 was north of 6 seconds. Six seconds. On a checkout page. During our biggest sale of the year.

That gap between average and P99 was the outage. It just wasn’t showing up anywhere our dashboards were looking.

Throughput: “How Much,” Not “How Fast”

Once we fixed the payment gateway call (turned out it wasn’t even latency-related — it was something worse, which we’ll get to), a second problem showed up almost immediately: our checkout service could only comfortably handle around 200 requests per second. Our flash sale traffic was spiking to 1,400 requests per second.

This is throughput. Not “is the response fast,” but:

How many requests can my system handle in a given window of time, without falling over?

Usually measured as TPS — transactions per second.

Here’s the mental model that finally made this click for me: imagine one cashier at a grocery store. That cashier might be very fast — 10 seconds per customer, excellent latency. But if 50 people are in line and only one register is open, doesn’t matter how fast that one cashier is. The line is the problem now. That’s a throughput problem, not a latency problem.

Sound familiar? Have you ever had a service that was individually fast, but still fell over under load? That’s this, every time.

How We Actually Scaled It

There are really only two levers here, and I’ve used both, depending on the budget I had at the time (because let’s be honest, infra costs money, and not every team has unlimited budget to throw servers at a problem).

Lever 1: Horizontal scaling — add more workers

                  ┌─────────────┐
 Client Requests →│Load Balancer│
                  └─────┬───────┘
        ┌───────┬───────┼───────┬───────┐
        ▼       ▼       ▼       ▼       ▼
     Server1 Server2 Server3 Server4 Server5
     (280 TPS total instead of 200 alone... wait, let's do it right)

If one checkout server handles 200 TPS, five identical servers behind a load balancer can theoretically handle 1,000 TPS. This is why cloud auto-scaling groups exist — spin up more boxes when traffic spikes, spin them down when it’s quiet. Simple, but not free, and not infinite.

Lever 2: Increase concurrency per server

If your servers are sitting at 15% CPU while only handling one request at a time, you’re leaving performance on the table. In Node.js this often means actually using your event loop properly instead of blocking it, or scaling with the built-in cluster module to use every CPU core on the box:

const cluster = require('cluster');
const numCPUs = require('os').cpus().length;
if (cluster.isPrimary) {
  console.log(`Spinning up ${numCPUs} workers`);
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
} else {
  require('./server.js'); // your actual express app
}

Why this matters: Node.js is single-threaded by default. One node server.js process uses exactly one CPU core, no matter how many cores your machine has. If you've got a 4-core server, 3 of those cores just sit idle while your app struggles under load. That's the "15% CPU while requests queue up" problem.

Picture it like this

                  Incoming requests (port 3000)
                            |
                    ┌───────┴────────┐
                    │  Primary process │  ← just manages workers,
                    │  (cluster.fork)  │     doesn't handle requests itself
                    └───────┬────────┘
        ┌───────────┬───────┴───────┬───────────┐
        ▼           ▼               ▼           ▼
    Worker 1     Worker 2       Worker 3     Worker 4
   (full Express app copy, running on its own core)

That one change, on a 4-core box, roughly quadrupled our checkout service’s throughput — no new servers, no extra cost.

But here’s the part that bit us: we got greedy and pushed CPU usage to 95%+ trying to squeeze out more throughput. Bad idea.

Once you cross roughly 80% sustained CPU, latency starts climbing fast — because requests start queueing instead of processing. Sound familiar? Yeah. Our “throughput fix” quietly turned back into a “latency problem.” These two are more connected than most articles give them credit for.

This is where “spare capacity” comes in.

At 50% CPU, your server is only using half its power on average. So when a burst hits — say traffic jumps 40% for a few seconds — there’s slack to absorb it. The extra requests get processed almost immediately using the spare capacity that was sitting idle.
At 95% CPU, there’s barely any slack left. When that same burst hits, there’s no spare capacity to absorb it — so the extra requests have nowhere to go except into a queue, waiting for a turn.

I keep my services capped around 70–80% peak CPU now. It’s not the most cost-efficient number on paper, but it’s the number that’s let me sleep through flash sales since.

So What Actually Happened That Night?

If you’re wondering what really caused the outage — it wasn’t one villain, it was both:

Our payment gateway call had no timeout set. When the third-party gateway slowed down (their problem, not ours), our requests just… waited. Forever, basically. That’s a latency problem — specifically, processing latency with no upper bound.
Because those payment requests were just hanging instead of timing out quickly, they kept holding onto a connection each. Soon all the connections were in use and stuck. So when new requests came in, there were no connections left for them to use. That’s what caused throughput to drop — and it all started from one latency bug (the missing timeout).

The fix was almost embarrassingly simple: add a 3-second timeout on the payment call, fail fast, retry once, and if it still fails, tell the user honestly instead of leaving them staring at a spinner. Throughput recovered instantly once requests stopped piling up.

Has this happened to you — one small missing timeout quietly taking down an entire system under load? I’d genuinely like to know I’m not the only one who’s learned this lesson the hard way. Drop it in the comments if you’ve got a blunder like this too.

What I Actually Took Away From This

Latency tells you how your system feels to one user. Throughput tells you how your system holds up under many users at once. You need both numbers, all the time, not just when things break — because by the time your dashboard turns red, your P99 users have already been suffering for a while.

If I could go back and give myself one piece of advice before that flash sale, it’d be this: stop trusting averages, stop assuming “green dashboard” means “happy users,” and always ask — fast for whom, and how many of them?

What does your monitoring setup actually track — averages, or percentiles? Genuinely curious how many teams are still flying blind on this one. Let me know below.