Prompt and context
The company wants to reduce SMTP STARTTLS downgrade and man-in-the-middle risk by publishing DNSSEC and TLSA records for its sending domain. Not every recipient supports DANE, and certificates rotate regularly. Design MX discovery, TLSA validation, compatibility fallback, rotation, monitoring, and incident rollback.
What the interviewer is testing
- Understanding that SMTP DANE depends on DNSSEC, the MX target, and TLSA, not only a certificate authority.
- Distinguishing opportunistic DANE, enforced TLSA, MTA-STS, and TLS reporting.
- Designing TLSA generation, TTL, certificate rotation, and DNSSEC failure handling.
- Including delivery success, downgrade, validation failure, and recipient compatibility in a canary.
Questions to clarify first
- Are you protecting the sending domain, receiving domain, or both? Does the sending MTA validate peers?
- Is the DNSSEC chain, resolver, and cache behavior reliable?
- What share of recipients support DANE, MTA-STS, or only STARTTLS?
- Are certificates CA-issued or self-signed, and what are rotation and rollback windows?
- On failure, may delivery be delayed, or must it fall back to plaintext?
A 30-second answer framework
The sending MTA resolves MX, validates TLSA through DNSSEC, and checks the peer certificate or key using the TLSA usage, selector, matching type, and association data. A DNSSEC failure or TLSA mismatch must not be treated as ordinary STARTTLS success; the policy should delay or reject delivery. Start in observation mode, canary domains, combine MTA-STS with TLS reports, and rotate with overlapping records and a rollback window.
Step-by-step deep answer
Step 1: Trace the protocol path
The sender obtains the transport host from the recipient domain's MX records and negotiates STARTTLS. DANE uses DNSSEC-protected TLSA records to bind the domain, port, and target certificate or key. It does not treat the MX name as the certificate name or bypass the TLS handshake and hostname rules.
Step 2: Choose TLSA semantics
TLSA certificate usage, selector, and matching type define what is matched. A record may match the full certificate, SubjectPublicKeyInfo, or a digest. A looser choice eases rotation; a tighter one provides stronger control. Records must represent the actual deployment chain and account for a backup key rather than copying one current fingerprint.
Step 3: Design DNSSEC and TTL
Verify the DNSSEC chain from the parent zone to TLSA and monitor signature expiry, DS changes, and resolution errors. TTL controls rotation propagation and revocation speed, so overlap records for the required window. Distinguish DNSSEC failure, SERVFAIL, and NXDOMAIN from “no TLSA” to avoid false downgrade after resolver or cache faults.
Step 4: Build compatibility policy
Validate TLSA for DANE-capable destinations. For others, apply a domain policy for MTA-STS, ordinary STARTTLS, or queueing. Enforcement must not silently fall back to plaintext. Put the failure reason, destination, DNSSEC state, and retry time into the queue and TLS report instead of one generic connection error.
Step 5: Rotate certificates and keys
Publish new and old TLSA records, deploy the new certificate or key, wait through TTL and an observation window, then remove the old record. Verify the view from several resolvers and MTAs at every step. For digest matching, record the tool, algorithm, and input. A failed rotation restores both old records and the old certificate.
Step 6: Canary and observe
Start with internal or low-risk recipient domains in observation mode, then gradually enforce. Monitor delivery success, TLSA matches, DNSSEC failures, certificate mismatches, queue age, and plaintext downgrades. Aggregate and redact TLS reports; do not send recipient addresses or complete message metadata to a third party.
Step 7: Exercise incidents and recovery
Drill expired DNSSEC signatures, bad TLSA publication, early certificate rotation, MX changes, resolver failures, and non-DANE peers. Freeze enforcement, extend queue retries, and roll back DNS or certificates; unfreeze only after independent MTAs and multiple public resolvers confirm recovery.
High-quality sample answer
I would make MX, DNSSEC, TLSA, and SMTP TLS separate observable steps: resolve MX, validate DNSSEC, then check the peer certificate or key using TLSA usage, selector, and matching type. A DNSSEC failure or TLSA mismatch leads to domain-policy rejection or queueing, never silent plaintext. Roll out in observation mode, canary enforcement, and combine MTA-STS with TLS reports. Publish overlapping TLSA records before the new certificate, wait through TTL, then remove the old record. Preserve DNS and certificate rollback paths and drill signature expiry, MX changes, and resolver failure.
Common mistakes
- Treating DANE as a CA check or MX-hostname check only.
- Continuing with ordinary STARTTLS or plaintext after DNSSEC validation fails.
- Removing the old TLSA record before deploying the replacement certificate.
- Testing with one resolver and sender and ignoring cache propagation.
- Including recipient addresses and full message metadata in TLS reports.
Follow-up questions and responses
Follow-up 1: How do DANE and MTA-STS differ?
DANE relies on DNSSEC and TLSA and fits MTAs that validate DNSSEC. MTA-STS publishes policy over HTTPS and relies on CA certificates. They can coexist; choose by peer capability and operating boundary rather than treating one as the other's silent fallback.
Follow-up 2: Does TLSA match a public key or a certificate?
Selector and matching type decide. A key or digest can reduce sensitivity to certificate-chain changes, but preserve the exact input, algorithm, and rotation record to avoid a wrong digest.
Follow-up 3: Can delivery fall back to plaintext after DNSSEC SERVFAIL?
Not when the domain policy requires DANE. Delay or reject and alert. Only an explicitly allowed compatibility policy can choose another path, and it must record the downgrade.
Follow-up 4: How do you roll back a bad TLSA publication?
Restore the old TLSA and certificate, wait for authoritative DNS, TTL caches, and multiple recursive resolvers to converge, then release the queue freeze. Keep the bad record, affected domains, and recovery time for audit.
Follow-up 5: How do you prove there was no plaintext downgrade?
Record TLS negotiation, TLSA state, and downgrade counts per destination domain and verify periodically from an independent MTA. Treat plaintext attempts as high-priority alerts and reconcile them with TLS reports and delivery logs.