How to Inject Network Latency with tc for Pre‑Prod Testing

Testing how an application behaves under poor network conditions is something I do regularly before pushing changes to live environments. Instead of guessing or relying on vague staging setups, I use the tc (traffic control) subsystem built into Linux to inject latency, jitter, and packet loss on demand. This lets me reproduce edge cases like high-latency cross-region links or congested last-mile connections without needing actual faulty hardware.

The core tool here is the tc command, part of the iproute2 suite. It works by attaching a queueing discipline (qdisc) to a network interface. The most common one for delay simulation is netem, which stands for "network emulator". With it, you can add fixed delay, variable delay (jitter), packet duplication, corruption, and more.

Let’s say I want to test how my web API handles 150ms of round-trip latency. I’d first identify the interface—usually eth0 or ens5 on a VM—and then apply a netem rule:

sudo tc qdisc add dev eth0 root netem delay 150ms

After this, any traffic leaving eth0 will experience an additional 150ms delay. I can verify it with ping from another host:

$ ping -c 5 10.0.0.5
PING 10.0.0.5 (10.0.0.5) 56 data bytes
64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=150.2 ms
64 bytes from 10.0.0.5: icmp_seq=2 ttl=64 time=150.1 ms
64 bytes from 10.0.0.5: icmp_seq=3 ttl=64 time=150.3 ms

That confirms the delay is active. But real networks aren’t perfectly uniform—they have jitter. To simulate that, I add a distribution. For example, 150ms ± 30ms with a normal distribution:

sudo tc qdisc change dev eth0 root netem delay 150ms 30ms distribution normal

Now the ping times will vary around 150ms, mimicking real-world variability. I’ve found this especially useful when testing timeout logic in microservices or retry mechanisms in client SDKs.

Sometimes I need to go beyond delay and test packet loss. Maybe my app runs over a flaky 4G link or a satellite connection. Adding loss is straightforward:

sudo tc qdisc change dev eth0 root netem delay 150ms 30ms loss 5%

This injects 5% random packet loss alongside the delayed, jittery link. I’ve used this combo to uncover bugs in HTTP clients that didn’t properly handle retransmissions or in video streaming adapters that failed to downgrade bitrate gracefully.

If I want to see what’s currently applied to an interface, I check with:

sudo tc qdisc show dev eth0

Output might look like:

qdisc netem 8001: root refcnt 2 limit 1000 delay 150.0ms 30.0ms loss 5% 0%

When I’m done testing, I clean up the rule to restore normal network behavior:

sudo tc qdisc del dev eth0 root

It’s important to run this cleanup—leftover tc rules can persist across reboots on some cloud images and cause confusing performance issues later. I’ve seen teams waste hours debugging "mysterious latency" only to find an old netem rule still active from a forgotten test.

For more complex scenarios, like simulating asymmetric links (different delay upstream vs downstream), I use ifb (Intermediate Functional Block) devices to shape traffic in both directions. But for most pre-prod validation—especially testing API timeouts, frontend loading behavior, or database connection pooling under stress—the simple egress delay on the server or client VM is sufficient.

I often combine this with monitoring tools like iftop, nload, or even application-level metrics (response times, error rates) to observe the impact in real time. As I mentioned before in my post about kernel CPU troubleshooting, having observability in place makes it easier to isolate whether slowdowns are due to injected network issues or something else.

One practical tip: if you’re testing inside a container, you’ll need to either run the container with --net=host or apply the tc rule from the host namespace targeting the veth pair. Docker and Kubernetes don’t let you modify tc inside the container by default due to privilege restrictions.

Finally, always test these changes in a non-production environment first. While tc is safe and reversible, applying heavy loss or delay on a live service can trigger cascading failures or false alarms in monitoring systems.

If you’re validating application resilience, simulating real network conditions with tc is one of the most effective, low-cost methods available. It turns abstract "what if" questions into concrete, repeatable tests.

Cover image: ₡ґǘșϯγ Ɗᶏ Ⱪᶅṏⱳդ · CC0 (Openverse / kamu malı) · https://www.flickr.com/photos/148598741@N02/53210652349