System design interview: How would you design a MASQUE CONNECT-UDP proxy?

Prompt and scope

A mobile client can reliably reach HTTPS, but voice, games, and real-time probes depend on UDP. Design a proxy that lets the client establish an HTTP tunnel to a target UDP host, covering lifecycle, forwarding, authorization, rate limits, DNS, failover, and fallback.

RFC 9298 defines CONNECT-UDP and a proxy template. RFC 9297 defines HTTP Datagrams and the Capsule Protocol for unreliable data and reliable control information respectively. A strong answer separates protocol semantics, proxy resources, and security policy; TLS alone does not prevent abuse.

What the interviewer evaluates

• Distinguish tunnel establishment, forwarding, and closure for CONNECT-UDP.
• Explain the boundary between HTTP/3 Datagrams and Capsules, including fallback.
• Design target allowlists, port policy, user authorization, and tenant quotas.
• Handle DNS, timeouts, retries, half-open sessions, and failover.
• Define bandwidth, concurrency, loss, latency, abuse, and cost metrics.
• State that a proxy cannot add UDP reliability, ordering, or end-to-end encryption semantics.

Clarifying questions

1. Do both client and proxy support HTTP/3 Datagrams, or must HTTP/2 work too?
2. Is the target an enterprise network, real-time media, or the open Internet? The risk model differs.
3. Does the client provide a hostname, and does the proxy resolve DNS? What are the privacy and lifetime requirements?
4. How many tunnels, bytes, and concurrent UDP flows may one tenant use? Is regional routing required?
5. What are the maximum idle period, packet size, and session duration?

30-second answer

I would restrict the proxy to authenticated tenants and an explicit target set, then establish a session with CONNECT-UDP. Capsules carry reliable control; HTTP/3 Datagrams carry UDP data when supported. Otherwise the proxy falls back to reliable encapsulation or rejects the request, without claiming equivalent latency. The data plane applies tenant token buckets, concurrency limits, and idle timeouts, while a controlled resolver binds DNS results to the session. Failover rebuilds only replayable state. I would validate p99 latency, loss, setup success, resource use, rejection rate, and abuse alerts.

Step-by-step design

1. Session and state machine

The client sends CONNECT-UDP for a target host and port. The proxy authenticates identity, policy, and quota, resolves the target, and creates a session. States should include pending authorization, connected, draining, and closed, each with a timeout and reason code so half-open sessions cannot hold resources indefinitely.

CONNECT / .well-known/masque/udp/example.test/443 HTTP/3
Host: proxy.example

An implementation must follow RFC 9298 request-template and encoding rules; the snippet only conveys intent.

2. Separate control and data planes

The Capsule Protocol suits reliable session control, errors, and close notifications. HTTP/3 Datagrams suit UDP data that does not require retransmission. Map each Datagram to the corresponding UDP socket and bound packet size, queue depth, and burst. If the path lacks Datagram support, explicitly choose reliable encapsulation or return unsupported instead of hiding the latency and CPU cost.

3. Authorization and target policy

Validate a short-lived credential, tenant state, and device binding before applying an allowlist by hostname, IP range, port, and purpose. Block loopback, private, metadata, and high-risk destinations. Resolve DNS with a controlled resolver, record the resolution version and TTL, and prevent re-resolution from silently crossing a tenant boundary.

4. Limits, quotas, and cost

Limit tunnels, packets per second, byte rate, session duration, and idle time per tenant. Use token buckets at both ingress and egress, with a global node safety threshold. Track proxy CPU, kernel sockets, queues, egress bandwidth, and per-session cost; limiting HTTP request count alone does not constrain a long-lived UDP flow.

5. Failure and fallback

Record distinct reasons for setup failure, unreachable targets, DNS timeout, and overload. New sessions may retry on a healthy node. UDP data already sent is generally unsafe to replay, so failover should restore control state or let the upper protocol establish a new session. If Datagrams are unsupported, choose reliable encapsulation or an explicit failure and measure the paths separately.

6. Observability and security operations

For each session record tenant, proxy node, policy version, setup and close reason, bytes, packets, estimated loss, p50/p95/p99 latency, and throttling events. Never log full credentials or sensitive payloads. Detect port scans, target concentration, amplification bursts, and cross-tenant contention, with rapid revocation by tenant, target, or region.

7. Rollout and acceptance

Canary fixed regions and allowlisted targets, comparing direct, Datagram, and fallback paths. Stress loss, reordering, proxy restart, DNS changes, half-open sessions, and excess traffic. Release gates should include setup success, real-time p99, loss, per-session resources, false rejection, and abuse-alert latency. Tighten policy or disable the path when a security or resource threshold is exceeded.

Sample strong answer

I would build an authenticated proxy limited to an explicit target set. After CONNECT-UDP, it validates credentials, target, and quota, resolves DNS through a controlled resolver, and creates a session. Capsules carry control; HTTP/3 Datagrams carry non-replayable UDP data when available. Each session has packet-size, queue, rate, idle, and lifetime limits.

Egress policy blocks private, loopback, metadata, and high-risk ports, while token buckets protect both directions. New sessions can fail over; sent UDP data is not replayed automatically. I would observe setup success, p99 latency, loss, resources, cost, scans, and amplification. Canary tests compare Datagram, fallback, and direct paths, with security and resource gates able to disable the feature.

Common mistakes

• Saying “put UDP inside HTTPS” without tunnel semantics → separate CONNECT-UDP, Capsules, and Datagrams.
• Treating the proxy as reliable transport → reliability and ordering remain an upper-layer concern.
• Allowing arbitrary destinations → use host and port allowlists and block special ranges.
• Limiting only HTTP requests → also cap tunnels, packets, bytes, lifetime, and idle time.
• Replaying all UDP data after failure → restore only replayable control state and let the upper layer reconnect.
• Measuring only average latency → include p99, loss, rejection, resource, and abuse metrics.

Follow-up questions and responses

What if HTTP/2 cannot carry HTTP/3 Datagrams?

Choose reliable encapsulation, a polling-style transport, or explicit rejection according to the workload. Measure latency, CPU, and bandwidth separately; do not claim equivalence with unreliable Datagrams.

How do you prevent SSRF?

Check resolved IPs after DNS and before connect, block loopback, link-local, private, metadata, and policy-external ranges, and handle DNS rebinding. Bind the policy version to the session and make it revocable.

Who retries lost UDP packets?

The proxy reports transport observations but does not replay application packets. An upper protocol with idempotency and sequence semantics decides retries; real-time media may drop or repair data instead.

How does a proxy restart recover sessions?

Prefer re-authentication and a new session. If control state must persist, restore only short-lived, verifiable state without non-replayable data, and drain old sockets immediately.

How do you show that limits do not cause false rejection?

Compare rejection and success by tenant, region, target, and client version. Set an error-rejection budget and replay bursts, long sessions, and node failures to verify quota behavior.