Benchmark: Docker Bridge vs. Host Network vs. CNI Overlay
Table of Contents
These days, most infrastructure runs on Kubernetes, but there are still some older services running as standalone containers. Every now and then, I come across an issue worth sharing with everyone.
I used to pay very little attention to container network modes. When a system was slow, we’d usually investigate the database first, then the cache, queries, and application code. Networking between services was often treated as something happening in the background. Sure, there was probably some overhead, but I never felt it would make a significant difference.
Then one day, during a quiet stretch of Jira duty, I had some free time and started looking at the p99 latency of an internal system.
The system showed no obvious signs of a bottleneck. CPU usage wasn’t high. The database wasn’t under pressure. Redis was doing fine too.
But whenever traffic increased, p99 would spike enough to become annoying. The system wasn’t completely down; everything just felt less responsive. Some unlucky requests would suddenly take noticeably longer.
At first, I thought it was probably the application.
Then I checked the connection pool, looked at GC, and investigated timeouts between services.
Eventually, I realized that the network path between services was also contributing quite a bit.
So I decided to benchmark it properly, comparing three common networking approaches:
- Docker bridge
- Host network
- CNI overlay
What I wanted to know was pretty simple: when services constantly exchange small requests, how significant is the difference in practice?
Especially at p95 and p99.
Benchmark Setup
The scenario I used was nothing unusual.
Service A calls Service B over internal HTTP. The JSON request payload is around 1KB, and the response is roughly the same size.
I removed TLS from the test to isolate the network overhead without mixing in encryption costs.
The receiving service also does very little work. It basically receives a request and immediately returns a response.
In other words, I wasn’t trying to benchmark frameworks or business logic.
I just wanted to isolate the networking cost between services.
I tested two scenarios.
The first was when both services were on the same node. This happens quite often in real environments, especially when the system is still relatively small or the scheduler places both workloads on the same machine.
The second was when the services were on different nodes.
This scenario is more representative of Production because once a system scales across several machines, cross-node communication becomes normal.
The test machines had the following specifications:
- 8 vCPU
- 16GB RAM
- 10GbE network
For each test, I ran a 10-second warm-up, measured for 60 seconds, repeated the test 5 times, and took the median.
The benchmark client ran on a separate machine to minimize interference between the client and the services being tested.
The Three Network Modes I Compared
Anyone who uses Docker is probably familiar with bridge networking.
It’s the default mode. Each container has its own network namespace, and traffic passes through the host’s network bridge, usually with NAT and conntrack involved along the way.
Host network is much simpler.
The container uses the host’s network stack directly. There are fewer layers and less processing, but the trade-off is weaker network isolation.
For overlay networking, I tested a common VXLAN-based setup used in multi-node clusters.
The advantage is a simpler networking model when running across multiple nodes.
The downside is that packets take a more complicated path and go through additional processing layers.
Results When Both Services Are on the Same Node
The first set of results was pretty much what I expected, although the differences weren’t particularly dramatic.
| Network mode | req/s | p50 | p95 | p99 |
|---|---|---|---|---|
| Host network | 61k | 0.42ms | 0.88ms | 1.60ms |
| Docker bridge | 56k | 0.48ms | 1.05ms | 2.10ms |
| CNI overlay | 51k | 0.57ms | 1.30ms | 2.70ms |
Host network performed best, which wasn’t surprising.
Bridge was slightly slower, with p99 increasing from 1.60ms to 2.10ms.
Overlay reached 2.70ms.
What I found interesting was that bridge actually performed much better than many people might expect.
Looking only at the same-node scenario, I don’t see much reason to avoid it.
It doesn’t perform as well as host networking, but it’s nowhere near bad enough to become a problem for most ordinary systems.
Overlay was starting to show its overhead.
Even so, if we only looked at the same-node results, it still wouldn’t seem particularly concerning.
The Difference Becomes Clearer Across Nodes
The differences became much more noticeable when requests had to travel between machines.
| Network mode | req/s | p50 | p95 | p99 |
|---|---|---|---|---|
| Host network | 39k | 0.95ms | 2.40ms | 4.80ms |
| Docker bridge | 35k | 1.12ms | 3.10ms | 6.90ms |
| CNI overlay | 30k | 1.45ms | 4.60ms | 10.80ms |
At this point, I almost stopped caring about p50.
The median latency still looked pretty reasonable. Everything was around 1ms to 1.5ms.
If you only looked at the median, it would be easy to conclude that network mode doesn’t make much difference.
But p99 tells a completely different story.
Host network was at 4.8ms.
Bridge increased to 6.9ms.
Overlay reached 10.8ms.
For a single hop, these numbers might still be acceptable.
But a microservices system rarely has just one hop.
A request might go through a gateway, authentication service, user service, and then another one or two supporting services.
The worst-case latency from each hop can add up very quickly.
That’s when the entire system starts feeling sluggish, even though no individual component appears to have a serious bottleneck.
I think this is what many teams are most likely to overlook.
Every service looks fine when examined individually.
But when the tail latency from multiple hops adds up, the result isn’t fine at all.
I Also Tested Burst Traffic
Production traffic doesn’t arrive at a perfectly steady rate, so I added another test.
Traffic doubled for a short period before returning to its normal level.
The results made the problem immediately obvious.
| Network mode | p50 | p95 | p99 |
|---|---|---|---|
| Host network | 1.20ms | 4.10ms | 7.90ms |
| Docker bridge | 1.40ms | 5.60ms | 11.80ms |
| CNI overlay | 1.85ms | 8.90ms | 18.70ms |
Under burst traffic, overlay networking’s disadvantages became much more apparent.
Its p99 was almost 19ms, while host networking remained below 8ms.
This was the point where I stopped thinking of network mode as a minor implementation detail.
If your services communicate frequently, exchange small requests, and operate with high concurrency, the network path can contribute much more to end-to-end latency than you might expect.
Why Does Overlay Have the Biggest Overhead?
To be fair, there’s a good reason why overlay networking is slower.
It’s solving a more complicated problem.
When you use an overlay network, you’re getting:
- A unified network across multiple nodes
- More flexible deployments
- Easier service and policy management
- Less complicated cluster scaling
In exchange, packets have to go through additional encapsulation, decapsulation, routing, policy processing, and several other steps.
With small requests, this kind of fixed overhead becomes noticeable very quickly.
It’s not necessarily a bandwidth problem.
The main reason is that the requests themselves are so short.
Adding even a little extra processing to every hop can make a visible difference.
Should We Use Host Networking for Everything?
I don’t think it makes sense to draw one conclusion and apply it everywhere.
Host networking clearly delivers the best latency in these tests.
But it also comes with several less pleasant trade-offs.
Port conflicts become more likely.
Network isolation is weaker.
And you lose some of the convenience of container networking.
Bridge sits somewhere in the middle and offers a pretty good balance.
If your system runs on only a few nodes or isn’t particularly sensitive to p99 latency, I think bridge is perfectly fine for many use cases.
Overlay naturally introduces more latency, especially for cross-node communication.
But when you’re running a multi-node cluster, that overhead may simply be the price you pay for operational flexibility.
At that point, a more useful question is often:
Which services are latency-sensitive enough to justify their own networking optimizations?
Rather than:
Should we get rid of overlay networking entirely?
A Few Takeaways from This Benchmark
The most memorable result wasn’t that host networking came out on top.
That was pretty easy to predict.
What stood out to me was that bridge networking wasn’t nearly as bad as I used to think.
And with overlay networking, the cost becomes most visible at p95 and p99, especially when services are on different nodes and traffic arrives in bursts.
If your system has only a few hops, generous timeouts, and isn’t especially sensitive to a few extra milliseconds, these differences may not become a real problem.
But if you’re running a bunch of services that constantly call each other, with fan-out and retries, and you’ve ever wondered why the system feels sluggish even though nothing seems completely bottlenecked, network mode is definitely something worth benchmarking.
Conclusion
After this experience, I no longer treat networking between services as something that just happens in the background.
Host networking still offers the best results if we’re looking purely at latency.
Bridge provides a pretty reasonable balance.
Overlay isn’t free, and the cost you pay usually shows up in p95 and p99 rather than p50.
Ultimately, there’s no single answer that works for every system.
Some environments need simplicity.
Others need flexibility.
And in some cases, shaving just a few milliseconds off tail latency can make a meaningful difference.
If you want to make the right decision, the best approach is still to benchmark your system using its real workload.
But at least after running these tests, I’ve come to realize that networking between services deserves much more attention than I used to give it.