9 min read
VI EN
Insights / Benchmark

Vertical vs. Horizontal Scaling Benchmark: Which Is More Practical for Production?

Nguyễn Thanh Tuấn
Published on Mar 3, 2026 • Cloud Platforms Architect
Benchmark Scale Dọc hay Ngang: Lựa chọn nào thực tế cho Production?
Table of Contents

I recently worked on a real-world task that I thought might help others with the same question, so I decided to write about it. It’s been ages since I last ran a benchmark 😀

Every now and then, someone on the team asks, “Why is the API fast sometimes and slow at other times?” Meanwhile, partners send over logs showing occasional timeouts.

Grafana doesn’t show anything alarming, but p95 and p99 keep creeping up, especially during peak hours.

It’s a pretty familiar scenario: most requests are still fine, but a small number take much longer to process. And those few requests are exactly what frustrate users. Retry rates start increasing, followed by occasional timeouts.

Based on my experience, I decided to try scaling the system.

At this point, we usually have two familiar options:

  • Vertical scaling: Upgrade a single instance’s resources, for example, from 2 vCPU to 4 vCPU and then 8 vCPU.
  • Horizontal scaling: Distribute the workload across multiple instances, for example, scaling from 1 to 2, 4, or 8 instances behind a Load Balancer.

Earlier this year, while catching up on some unfinished tasks and reading posts shared by others, I decided to run a benchmark myself… (This article also borrows some writing formats from other authors on devops.vn.)

A quick note: The numbers below come from a lab environment, so your real-world results may differ. However, the general trends are often quite similar if you follow the right methodology and avoid operating system limitations.

The Problem I Wanted to Benchmark

I’ve seen plenty of benchmarks where a backend returns OK almost instantly, everything runs on the same machine, and all heavy workloads are disabled. Then someone concludes that one approach is twice as fast as another.

Those results might be interesting to look at, but they aren’t particularly useful in real-world environments.

So I chose a request pattern that actual services commonly handle:

  • Small JSON payloads, with requests and responses around 1 to 3 KB.
  • Primarily using Keep-alive connections.
  • The application performs real processing: parsing JSON, doing lightweight authentication checks, and marshaling responses.
  • One external operation, either a Redis cache hit or a very lightweight database query, to simulate a more realistic workload while keeping the focus on application scaling.

I focused on three key metrics: (RPS)[https://devops.vn/posts/rps-la-gi-trong-devops/], (error Rate)[https://devops.vn/posts/error-rate-la-gi-doc-trong-he-thong-web/], and especially p95/p99.

If RPS looks great but p99 is terrible or the error rate increases, I consider the result unacceptable.

To make p99 easier to visualize: out of 10,000 requests, roughly 100 requests fall into the p99 tail.

That’s not many, but it’s more than enough to leave users with a bad impression.

Environment Setup

I separated the environment into three components to avoid the results being affected by everything running together over loopback:

  • Loadgen: Used wrk2 to generate a steady load.
  • Load Balancer: Used HAProxy.
  • App instances: All instances had identical configurations.

Everything ran within the same internal network (LAN/VPC) to keep internal latency below 1ms.

This allowed me to focus on measuring the scaling strategy rather than Internet network latency.

Rules for Getting Objective Results

To avoid misleadingly good benchmark numbers, I followed these rules:

  • Disable or minimize access logging. Excessive logging can create a CPU bottleneck during benchmarks.
  • Increase ulimit -n and make sure the backlog is large enough to avoid operating system limitations.
  • Run separate warm-up tests and only collect results once the system reaches a stable state.
  • If latency looks good but errors or retries increase, consider the result invalid and investigate again.

The Two Scaling Approaches I Compared

Vertical Scaling

  • 1 instance with 2 vCPU
  • 1 instance with 4 vCPU
  • 1 instance with 8 vCPU

Horizontal Scaling

  • 2 instances, each with 2 vCPU
  • 4 instances, each with 2 vCPU
  • 8 instances, each with 2 vCPU

For the main comparisons, I balanced the total vCPU resources.

For example, I compared one 8-vCPU instance against four 2-vCPU instances, since this is also how many people evaluate costs.

I kept the eight-instance configuration to examine the overhead of scaling horizontally to a larger number of instances. I don’t consider it a resource-equivalent comparison with the 8-vCPU configuration.

Test Scenarios

Steady Load

  • Gradually increase RPS until the system reaches its stable throughput limit.
  • Warm up for 15 seconds.
  • Measure for 60 seconds.
  • Collect p95, p99, error rate, CPU, and memory metrics.

Each configuration was tested three times, and I used the median result to keep the comparison objective.

Spike Traffic

  • Baseline of 8k RPS.
  • Suddenly increase to 25k RPS for 60 seconds.
  • Observe how much p99 increases and how quickly the system recovers.

Steady Load Results

I defined the stable throughput threshold using these criteria: p99 below 100 ms and error rate below 0.5%.

Vertical Scaling

Configuration Max Stable RPS p95 p99 App CPU Notes
1 instance, 2 vCPU ~6.5k ~18 ms ~85 ms ~90–95% Hits the CPU ceiling early
1 instance, 4 vCPU ~12k ~16 ms ~78 ms ~85–90% Suitable for moderate workloads
1 instance, 8 vCPU ~21k ~14 ms ~70 ms ~80–88% More stable p99 with less variation

Horizontal Scaling

Configuration Max Stable RPS p95 p99 Total App CPU Load Balancer CPU Notes
2 instances, 2 vCPU each ~11k ~17 ms ~82 ms ~170–185% ~10–15% Close to 1 instance with 4 vCPU
4 instances, 2 vCPU each ~20k ~16 ms ~76 ms ~340–380% ~18–25% Close to 1 instance with 8 vCPU
8 instances, 2 vCPU each ~30k ~18 ms ~95 ms ~650–720% ~35–45% Overhead starts becoming noticeable

For total CPU usage, I use the convention that 100% represents one fully utilized CPU core.

So ~650% means approximately 6.5 cores are busy.

Looking at the steady load results, two things stand out to me:

  • When total compute resources are equivalent, vertical and horizontal scaling deliver fairly similar results. Increasing the number of instances doesn’t automatically make things better.
  • As you scale horizontally to more instances, the operational overhead starts becoming noticeable, especially in p99, because the load balancer has more work to do and there are more connections to manage.

Spike Test

Case A: 1 Instance with 8 vCPU

  • Application CPU quickly reaches ~95–100%.
  • p99 jumps to 180–320 ms because requests start queueing.
  • Error rate reaches 0.8–1.8%, with occasional timeouts.
  • The system recovers quickly once the spike ends, but users clearly experience slower responses during the spike.

Case B: 4 Fixed Instances with 2 vCPU Each, Without Autoscaling

  • Total compute resources are equivalent to 8 vCPU.
  • p99 is around 160–280 ms.
  • Error rate ranges from 0.5–1.2%.

Case C: Autoscaling from 4 to 8 Instances with 2 vCPU Each

  • Assumed scaling and readiness time: 30–60 seconds.
  • Results remain poor during the first 20–30 seconds because the additional instances aren’t ready yet.
  • Once scaling completes, p99 drops to around 80–120 ms, and the error rate becomes almost zero.

This part highlights a very practical point:

Autoscaling is only effective when a traffic spike lasts long enough for the system to add instances and stabilize.

If the spike is too short, there will still be a period of degraded performance that’s difficult to avoid.

Mistakes I’ve Made While Benchmarking

To be fair, my first attempt also failed in a few areas.

I’m listing them here so you can avoid unnecessary measurement errors and save yourself some time :))

  1. I forgot to disable access logs, which reduced performance and increased CPU usage. I initially thought the application itself was underpowered.
  2. I tested an overly simple backend and got unrealistically good results. It was obvious that the test didn’t represent a real workload.
  3. I didn’t distinguish between keep-alive and short-lived connections. Simply changing the client’s connection behavior could completely reverse the results.
  4. I forgot to increase ulimit -n. As concurrency increased, connection errors started appearing, and I almost blamed the system for being unable to handle the load.

Running a benchmark isn’t difficult.

The difficult part is making sure the benchmark results actually reflect real-world conditions.

Factors That Can Change the Conclusion in Production

Horizontal scaling can cause problems if you overlook these factors:

  • Connection storms hitting the database or Redis: More instances mean more connections. Without proper connection pooling and throttling, the bottleneck can quickly shift to the database.
  • Cache duplication: More instances mean more separate in-memory caches. If cache hit rate is important, vertical scaling can sometimes be more beneficial because the cache remains centralized.
  • Uneven traffic distribution: Keep-alive connections or skewed hashing can cause some instances to handle more traffic than others, leading to poor p99 even when total CPU capacity is still available.
  • GC pauses or thread pool bottlenecks: CPU usage may not be maxed out, but p99 can still increase significantly because the system is bottlenecked elsewhere.

Final Takeaways for Real-World Use

Ultimately, the right scaling approach depends on the problem you’re solving and the actual conditions of your system.

Prioritize vertical scaling when:

  • Your system is small to medium-sized and traffic is relatively stable.
  • You want simpler operations and fewer potential points of failure.
  • You don’t have a mature autoscaling setup yet, or you don’t want to introduce more system complexity.

Move toward horizontal scaling when:

  • You need to ensure high availability (HA).
  • Traffic spikes happen frequently.
  • You have good control over your load balancer, health checks, and autoscaling.
  • Connection pooling and caching are properly managed so that the additional load doesn’t simply shift the bottleneck to the database.

Share this article

Theo dõi
Thông báo của
0 Bình luận
Phù hợp nhất
Mới nhất Cũ nhất
Post link copied to clipboard!