Prompt
Design browser-side end-to-end encryption for a multi-party WebRTC meeting. The media server forwards RTP but must not read audio or video. The sender and receiver process encoded frames in workers. Explain RTCRtpScriptTransform, RTCRtpScriptTransformer, key distribution, keyframes, performance, failure recovery, and compatibility. Distinguish encoded-frame transforms from transport TLS.
What the interviewer is testing
The test is whether you understand the browser media-pipeline boundary: transform after encoding and before decoding, preserve frame order from readable to writable, move per-frame crypto to a worker, and handle key rotation, join-time keyframes, dropped frames, and unsupported browsers. Saying “encrypt RTP” without a frame lifecycle is incomplete.
Clarifying questions
- Which parties must be protected from the media server, and may it only forward packets?
- Is audio included, along with screen share and recording?
- Are keys distributed by authenticated end-to-end signaling, and how are departed members revoked?
- What browser matrix is supported? Should unsupported clients be rejected, downgraded, or have E2EE disabled?
A 30-second framework
TLS protects transport links; it does not prevent a media server from seeing plaintext. E2EE encrypts after the sender encoder and decrypts before the receiver decoder. Cover five layers: capability detection, the worker frame pipeline, keys and nonces, keyframes and retry, and downgrade/observability. MDN marks the API Baseline 2025, but the browser matrix still needs testing.
Step-by-step design
1. Detect capability and attach early
Construct RTCRtpScriptTransform with a worker, a direction marker, and transferable MessagePort. Attach it to RTCRtpSender.transform on the sender and RTCRtpReceiver.transform on the receiver before the first frame. A failed capability check is an explicit state, never a silent success with plaintext media.
2. Process frames in a worker
The worker handles rtctransform, reads encoded frames from event.transformer.readable, runs a TransformStream, and writes to event.transformer.writable. Preserve order and enqueue each frame exactly once; close the stream and report state on errors. The main thread sends configuration and short-lived key handles, not per-frame crypto work.
3. Define ciphertext and key lifetimes
Create a context per meeting, sender, and key epoch. Derive a never-reused nonce from a frame counter plus the stream identity. Include version, epoch, and an authentication tag in the ciphertext, and verify before decrypting. Distribute keys over authenticated end-to-end signaling, allow a short dual-epoch overlap during rotation, and revoke new-frame access when a member leaves.
4. Recover with keyframes
A new participant may receive a delta frame before a keyframe and cannot decode it. A receiver transform can call sendKeyFrameRequest() after a new key arrives or decoding becomes impossible; a sender transform can call generateKeyFrame(). Both return promises, so check direction and video state and rate-limit requests.
5. Budget latency and memory
Minimize copies and garbage collection, reuse frame buffers where safe, and keep worker concurrency bounded. Measure transform latency, queue depth, drop rate, and keyframe-request rate. If crypto exceeds the latency budget, lower video quality or pause a track rather than blocking the UI thread.
6. Handle errors and reconnects
Expired keys, authentication failures, worker crashes, and rejected API calls enter an observable state machine. Restart a worker for transient failures and resume the current epoch; stop the track and explain the state when recovery fails. After PeerConnection renegotiation, attach a new transform instead of assuming the old worker follows the new sender.
7. Set compatibility and security boundaries
MDN labels Encoded Transform Baseline 2025, yet older browsers and devices may lack it. Product policy must explicitly reject, disable E2EE, or permit a trusted media server to transcode. The W3C document is a Working Draft, so interface stability and implementation differences must remain visible in the rollout plan.
Example of a strong answer
“I would attach RTCRtpScriptTransform after the sender encoder and before the receiver decoder. A worker reads encoded frames from readable, applies authenticated encryption, and writes to writable. End-to-end signaling manages keys per meeting, member, and epoch; a never-reused frame counter forms the nonce, and the receiver verifies the tag before decrypting. When a new member or key cannot decode a delta frame, the receiver rate-limits sendKeyFrameRequest and the sender can generateKeyFrame. We measure transform latency, queue depth, drops, and authentication failures; failed recovery stops the track. Capability detection chooses rejection, downgrade, or disabled E2EE, and the rollout records Baseline 2025 plus the W3C Working Draft status.”
Common failure modes
- Treating TLS as media end-to-end encryption.
- Encrypting every frame on the main thread or transforming raw frames instead of encoded frames.
- Reusing nonces, skipping authentication checks, or omitting epochs and revocation.
- Ignoring keyframes, worker crashes, reconnects, and browser differences.
- Claiming WebRTC support without checking the Encoded Transform interfaces.
Follow-up directions
Why transform after encoding?
Encoded frames are smaller and remain in the RTP pipeline; transforming raw frames adds copies and computation before encoding.
Will sendKeyFrameRequest() always send a request?
No. The user agent may decide it is unnecessary while still fulfilling the promise, so the product needs waiting and timeout handling.
How should keys reach the worker?
Pass short-lived handles through options or a transferable MessageChannel; avoid exposing long-lived keys to unrelated scripts.
Can unsupported browsers silently downgrade?
No. The user and meeting policy must clearly state whether the current media is E2EE.
How do you prove the server sees no plaintext?
Capture data at the forwarding node in a test environment and verify only ciphertext frames are visible; audit signaling, workers, key services, and recording permissions.
References
MDN “Using WebRTC Encoded Transforms”, MDN “RTCRtpScriptTransformer”, and the W3C “WebRTC Encoded Transform” Working Draft.