How I sample DNS latency without lying to myself

Comparing DNS resolvers sounds easy until the numbers refuse to repeat. This post is the methodology behind the lab experiment, written down so the next iteration does not rediscover the same traps.

The traps

Warm caches pretend to be fast resolvers. I flush between probes and separate uncached-first-query from steady-state. Clock drift ruins percentiles: everything is stamped against an NTP-synced clock, and probes run on the same host so differences cancel.

Order bias is the sneaky one. Running resolver A's probes first every round means A wins the warmed-up slots. The scheduler round-robins and shuffles; four hundred thousand samples later, the ordering effect is below measurement noise.

What the data showed

For a home connection, resolver choice moved the median by single-digit milliseconds and the p95 by more, dominated by one provider's occasional slow answer. Retries mattered more than medians.

Raw samples are published on the lab site as JSON for anyone who wants to argue with the numbers. That is the point of writing the method down: conclusions can be checked.