Using Linux tc netem to Add Jitter for Realistic WAN Simulation

I’ve seen too many apps pass lab tests only to choke when real users hit them over unpredictable links. The fix? Don’t guess—simulate. In my environment, I use Linux tc netem to inject controlled jitter, delay, and loss into test traffic before anything touches production. It’s not about breaking things; it’s about finding where they’ll break first.

Start by identifying the interface carrying your test traffic. Usually it’s a veth pair, bridge, or dedicated NIC on your test host. Run ip link show to confirm. Once you know the device—say eth1—you can shape it.

First, add a base delay with variability. This mimics real WAN where RTT isn’t fixed:

sudo tc qdisc add dev eth1 root netem delay 100ms 30ms distribution normal

Here, 100ms is the average delay, 30ms is the standard deviation (jitter), and normal gives a Gaussian spread—far more realistic than uniform random. You’ll see most packets cluster around 100ms±30ms, with rare outliers. That’s what actual broadband or mobile links feel like.

To verify, ping from another host:

ping -c 10 192.168.100.5

Look at the min/avg/max/mdev line. The mdev (mean deviation) should hover near your jitter value. If it’s too low, increase the jitter parameter; too high, and you risk false positives.

Sometimes you need loss too—especially for testing timeouts or retries. Combine delay and loss:

sudo tc qdisc change dev eth1 root netem delay 100ms 30ms loss 2%

Now 2% of packets are dropped, on top of the jittered delay. This combo surfaces issues like stalled connections, failed health checks, or cascading timeouts in microservices.

Want to test bursty loss? Use the correlation option:

sudo tc qdisc change dev eth1 root netem delay 100ms 30ms loss 2% 25%

The second loss percentage is the probability that a lost packet follows another loss—creating bursts. This is nastier and closer to real wireless or congested links.

When you’re done, clear the qdisc:

sudo tc qdisc del dev eth1 root

Always clean up. Leaving netem on a dev interface can baffle colleagues hours later.

I use this before rolling out API gateway changes or database migration scripts. As I mentioned before in my post on latency SLOs in Argo Rollouts, simulating delay helps validate SLO-based canaries better than error-rate alone. If your app degrades gracefully under 150ms p95 latency with jitter, it’s more likely to survive a real WAN hiccup.

One warning: netem adds kernel overhead. On busy test hosts, monitor CPU—especially if shaping gigabit+ traffic. For high rates, consider offloading to a dedicated appliance or using tc with htb for rate limiting first, then netem on top.

For recurring tests, I script it. A simple bash function wraps the tc commands, takes interface, delay, jitter, loss as args, and ensures cleanup on exit. Pair it with your test suite—run the simulation, hit the app with load, then tear down.

This isn’t about perfect fidelity. It’s about catching the obvious fragility early: hardcoded timeouts, missing retries, UI timeouts that assume LAN speeds. If your app works with 200ms RTT and 50ms jitter, it’ll handle most branch offices or remote users. Test it there first.

Cover image: Unknown · CC0 (Openverse / kamu malı) · https://www.rawpixel.com/image/6038427/photo-image-public-domain-technology-line