ResolverLab: High-Performance DNS Benchmarking Tool

← Back to Blog

Introduction

In today's privacy-conscious internet landscape, DNS resolvers have become critical infrastructure. From ad-blocking capabilities to malware protection and privacy preservation, choosing the right DNS service directly impacts your browsing experience, security posture, and data privacy. But how do you objectively compare 50+ DNS providers across multiple protocols and performance metrics?

ResolverLab is a high-performance Python analytics suite I built to answer exactly this question. It's designed to benchmark filtered DNS services across a comprehensive set of metrics: block accuracy, latency distributions, false positive rates, cache behavior, and protocol-specific performance (UDP, TCP, DoH, DoT).

                💡 Core Mission: Provide data-driven insights to help users, network administrators,
                and privacy advocates make informed decisions about their DNS infrastructure.
            

The Problem: DNS Provider Overload

The modern DNS ecosystem is fragmented. Major providers like Cloudflare, Google, Quad9, AdGuard, NextDNS, and dozens of others each claim superior performance and filtering capabilities. However:

Inconsistent Metrics: Providers report different performance characteristics with varying measurement methodologies.
Filtering Gaps: Ad-blocking effectiveness varies wildly between services and block list categories.
Protocol Differences: UDP, DoH, and DoT have distinct performance profiles that aren't clearly documented.
Cache Behavior: DNS cache hit rates and TTL handling significantly impact real-world latency but are rarely measured.
False Positives: Over-aggressive filtering can break legitimate websites, but these failures are underreported.

Without a unified benchmarking framework, comparing DNS services becomes an exercise in marketing claims rather than empirical data.

The Solution: ResolverLab Architecture

ResolverLab addresses these challenges through a modular, async-first architecture designed for high-concurrency DNS testing. Here's how the system is structured:

1. Multi-Protocol DNS Engine

The core DNS engine supports four protocols with native implementations:

UDP (Port 53): Traditional DNS with minimal overhead.
TCP (Port 53): Reliable transport for larger responses and fallback scenarios.
DNS over HTTPS (DoH): Encrypted DNS using HTTPS transport, preventing ISP surveillance.
DNS over TLS (DoT): Encrypted DNS on port 853, optimized for privacy with lower overhead than DoH.

async def query_dns(resolver, domain, protocol='UDP', query_type='A'):
    """
    Unified async DNS query interface supporting UDP, TCP, DoH, DoT
    """
    if protocol == 'DoH':
        return await query_doh(resolver['url'], domain, query_type)
    elif protocol == 'DoT':
        return await query_dot(resolver['ip'], domain, query_type)
    elif protocol == 'TCP':
        return await query_tcp(resolver['ip'], domain, query_type)
    else:  # UDP
        return await query_udp(resolver['ip'], domain, query_type)

2. Automated Blocklist Integration

ResolverLab automatically fetches and categorizes blocklists from leading sources:

StevenBlack: Unified hosts files combining ads, malware, and tracking domains.
hagezi: Multi-category threat intelligence (ads, phishing, cryptomining).
OISD: Optimized and deduplicated blocklists.
AdGuard: Proprietary filtering rules optimized for DNS-level blocking.
Phishing Army: Real-time phishing domain detection.

# Fetch blocklists for specific categories
python dns_benchmark.py fetch blocklists --categories ads malware

# Or fetch all blocklists
python dns_benchmark.py fetch blocklists --all

3. Comprehensive Metrics Framework

Every DNS query generates rich telemetry data that feeds into a multi-dimensional analysis:

Block Rate Blocked / Total Queries

False Positive Rate FP / (FP + TN)

Precision TP / (TP + FP)

F1 Score 2 * (Precision * Recall) / (P + R)

Latency Percentiles: P50, P95, P99 response times to capture tail latency.
Cache Hit Rates: Measure DNS cache efficiency and TTL adherence.
Error Rates: SERVFAIL, NXDOMAIN, timeout distribution analysis.
Protocol Overhead: Compare UDP vs DoH vs DoT handshake and transport costs.

4. Async Query Engine

To benchmark 50+ DNS services against hundreds of domains without waiting hours, ResolverLab leverages Python's asyncio for concurrent DNS queries:

async def benchmark_services(services, domains, protocol='UDP'):
    """
    Concurrently benchmark multiple DNS services
    with per-service rate limiting
    """
    tasks = []
    for service in services:
        limiter = RateLimiter(max_rate=100)  # 100 queries/sec
        task = asyncio.create_task(
            test_service(service, domains, protocol, limiter)
        )
        tasks.append(task)
    
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

This architecture enables testing tens of thousands of DNS queries in minutes rather than hours, while respecting provider rate limits to avoid getting blocked.

Real-World Benchmarking Workflows

ResolverLab is built around practical use cases. Here are three common workflows:

Workflow 1: Compare Ad-Blocking Effectiveness

# Test top ad-blocking DNS services
python dns_benchmark.py fetch blocklists --categories ads
python dns_benchmark.py test \
  --categories ads \
  --domains 200 \
  --services "AdGuard DNS" "NextDNS" "Mullvad Ad Block"

This workflow compares how effectively each service blocks advertising domains from curated blocklists. The generated report includes radar charts showing multi-metric comparisons across block rate, precision, and latency.

Workflow 2: Protocol Performance Battle

# Compare UDP vs DoH vs DoT for Cloudflare
python dns_benchmark.py test \
  --services "Cloudflare DNS" "Cloudflare DoH" "Cloudflare DoT" \
  --domains 100

This benchmark reveals the performance trade-offs between protocols. While DoH and DoT provide encryption, they introduce TLS handshake overhead. The report visualizes latency distributions and success rates across protocols.

Workflow 3: Full Stress Test

# Test ALL services against ALL blocklists
python dns_benchmark.py fetch blocklists --all
python dns_benchmark.py test --all --domains 50
python dns_benchmark.py report generate --latest --format html json csv

                ⚠️ Warning: This scan performs thousands of queries and may take 10-20+ minutes
                depending on your network connection and DNS service response times.
            

Visualization & Reporting

Raw data is useful, but actionable insights require visualization. ResolverLab generates interactive HTML reports using Plotly:

Radar Charts: Multi-dimensional comparison across block rate, precision, recall, F1 score, and latency.
Protocol Comparison: Side-by-side performance metrics for UDP, TCP, DoH, and DoT.
Latency Distribution: Box plots and histograms showing P50/P95/P99 response times.
Error Analysis: Breakdown of SERVFAIL, NXDOMAIN, and timeout rates.
Cache Performance: Cache hit rates and latency improvements from caching.

Reports can be exported in multiple formats: HTML (interactive), JSON (programmatic access), and CSV (spreadsheet analysis).

Technical Challenges & Solutions

Challenge 1: DNS-over-HTTPS Implementation

DoH requires sending DNS queries as HTTPS POST/GET requests with specific wire-format encoding. Unlike UDP, there's no standard Python library for DoH queries.

Solution: Implemented a custom DoH client using aiohttp with DNS message encoding via dns.message:

import dns.message
import aiohttp

async def query_doh(url, domain, query_type='A'):
    # Build DNS query message
    query = dns.message.make_query(domain, query_type)
    wire_format = query.to_wire()
    
    # Send DoH POST request
    async with aiohttp.ClientSession() as session:
        async with session.post(
            url,
            data=wire_format,
            headers={'Content-Type': 'application/dns-message'},
            timeout=aiohttp.ClientTimeout(total=5)
        ) as resp:
            response_data = await resp.read()
            response = dns.message.from_wire(response_data)
            return response

Challenge 2: Rate Limiting & Fair Testing

Bombarding DNS services with thousands of concurrent queries can trigger rate limiting or IP bans.

Solution: Implemented per-service rate limiters using token bucket algorithm, respecting each provider's documented query limits.

Challenge 3: Cache Contamination

Testing DNS caching requires careful ordering: if you query the same domain twice, the second query will be cached, skewing latency results.

Solution: Implemented a "cache warmup" mode that pre-queries domains, followed by a second round to measure cache hit performance separately.

Performance Results

Running ResolverLab on a production network with 50 DNS services and 500 test domains reveals interesting patterns:

~15-30ms UDP P50 Latency

~40-70ms DoH P50 Latency

85-95% Top DNS Cache Hit Rate

2-8% False Positive Range

UDP remains fastest: Traditional UDP queries consistently show 2-3x lower latency than encrypted protocols.
DoT outperforms DoH: DNS over TLS has ~15-20% lower latency than DoH due to reduced transport overhead.
Cache efficiency varies: Premium DNS services like Cloudflare and Google achieve 90%+ cache hit rates, while smaller providers often struggle with cache coherency.
False positives are real: Aggressive ad-blocking DNS services can have 5-10% false positive rates, breaking legitimate analytics and CDN domains.

Key Takeaways

Protocol Choice Matters: If privacy is paramount, use DoT. If you need maximum compatibility, stick with UDP/TCP.
Test Before Deploying: DNS filtering can break legitimate services—always benchmark with your actual domain patterns.
Cache Performance is Critical: In production environments, cache hit rates matter more than cold-start latency.
No One-Size-Fits-All: The "best" DNS service depends on your priorities: privacy, speed, filtering accuracy, or geographic proximity.

Future Enhancements

ResolverLab is actively evolving. Planned features include:

DNSSEC Validation: Verify cryptographic signatures to detect DNS spoofing.
Geographic Testing: Distributed probes to test DNS latency from multiple regions.
Machine Learning: Classify DNS providers based on filtering philosophy and performance profiles.
Real-Time Monitoring: Continuous DNS health checks with alerting for latency spikes or outages.

Explore the Source Code: The complete benchmarking suite is open-source on GitHub:

👉 github.com/DNSdecoded/ResolverLab

Interested in DNS infrastructure, performance engineering, or network security? Let's connect!

← Back to Blog