Prompt and context
A platform accepts text, images, audio, and video. Users expect posts to become visible quickly, while the platform must act before high-risk content spreads widely. Design upload pre-screening, asynchronous deep analysis, human review, appeals, post-publication rescanning, and policy iteration. You do not need to train a model, but you must explain how model outputs become auditable business decisions.
Public system-design interview material frames this problem as a layered pipeline with confidence bands, a human queue, and an appeal path. For a recomputable capacity assumption, start with 1,000,000 uploads per hour, about 278 requests per second on average and about 2,778 at a 10x peak; synchronous pre-screening may add at most 100 ms to publishing, and reviewers can handle 100,000 cases per day. Replace these numbers when the interviewer provides different ones.
What the interviewer is testing
The interviewer wants to see whether you split moderation into decisions with different latency and risk, instead of drawing one classifier. A strong answer separates immediate blocking, publish-then-rescan, and human review; sets per-policy thresholds; records model versions and evidence; and explains how an incorrect decision is corrected.
System-design signals also include peak capacity, queue age, duplicate work, model outages, adversarial inputs, reviewer safety, appeal deadlines, and tenant or regional isolation. Accuracy alone does not provide online governance without safe defaults, replayable evidence, and operational metrics.
Clarifications to ask first
What must be blocked
Ask whether every item needs pre-publication moderation or only clearly high-risk categories must be blocked synchronously. With a roughly 100 ms synchronous budget, deep video analysis belongs in an asynchronous path; if a category must be removed with no delay, reserve a heavier synchronous check for it.
How should error costs be ordered
Ask whether removing legitimate speech or missing high-harm content is more costly, and whether the tradeoff changes by country, age, or policy category. The answer determines thresholds, the size of the human band, and whether an automated action may delete content directly.
How long must review evidence live
Clarify appeal windows, regulatory evidence, media retention, and deletion requirements. If a decision must be replayed, store the policy and model versions, input digest, scores, cited evidence, and reviewer action instead of only the final label.
What are the model and reviewer limits
Ask whether publishing may degrade when a model is unavailable, how many cases reviewers can process each day, and the maximum wait. When capacity is short, reduce low-risk asynchronous admission or delay non-critical work; never let an unbounded queue silently release high-risk content.
A 30-second answer
“I would split the path into synchronous pre-screening and asynchronous deep review. On upload, run hash matches, format and size checks, a lightweight classifier, and baseline policy checks. Block clear high-risk results, send the confidence band to a human queue ordered by risk and expected exposure, and publish low-risk content with a post-publication rescan marker. Every decision stores its model, policy, evidence, and version, while appeals go to a different reviewer. I would scale pre-screening for the 2,778-request-per-second peak, isolate queues by tenant and risk, and monitor misses, false positives, queue age, exposure time, and reviewer load. Model or queue failures use a safe default and an explicit human fallback.”
Step-by-step deep dive
Step 1: Split synchronous and asynchronous decisions
After accepting content, the ingress writes an immutable contentId and media reference. The synchronous layer performs fast hash matching, format and size checks, a lightweight text or image model, and regional policy checks. A clear high-risk result is blocked; an uncertain result enters PENDING_REVIEW; a low-risk result may publish with a rescan marker. Video frame sampling, speech recognition, and multi-model fusion run asynchronously so deep analysis does not block every post.
Step 2: Act on categories, not one global score
Do not use one global threshold for every policy category. A high-harm category may use a lower automatic-block threshold and send borderline cases to people; a strongly contextual category may use a wider human band to reduce false positives. Threshold configuration carries a policy version and region, and the decision event stores the raw score and threshold so the action can be explained later.
Step 3: Build a scalable review queue
Write review tasks to a durable queue and calculate priority from risk, expected exposure, user reports, and deadline. Each task carries tenant or region, content version, policy version, and a lease; a reviewer holds it briefly, and expiry returns it to the queue. Shard by category and region so one popular category cannot exhaust every reviewer. When queue age crosses a threshold, reduce admission for low-risk asynchronous work and alert.
Step 4: Make appeals and feedback replayable
An appeal creates a new case instead of overwriting the original decision. The second reviewer sees the original evidence, policy, and model versions with a different permission set. An overturn restores or keeps the restriction and emits an audit event. Sampled human outcomes, appeal reversals, and new evasion examples feed an evaluation set, not unreviewed training data that could amplify bad labels.
Step 5: Observe risk, experience, and cost
Immediate metrics include pre-screen latency, block rate per category, asynchronous queue length and age, model errors, reviewer throughput, and service errors. Once labels mature, compute precision, recall, false positives, false negatives, and appeal reversals by category and region. Also measure high-risk exposure count multiplied by online time, reviewer exposure, and cost per item; model accuracy alone cannot reveal governance failure.
Step 6: Cover failures and adversarial inputs
When a model times out or is unavailable, retain hash and rule blocking. Unknown-risk content goes to restricted publishing or people; “no result” is not equivalent to safe. Rate-limit and budget re-encoded media, obfuscated text, duplicate uploads, and attempts to manufacture queue load. Every automated action must be replayable by contentId for incident analysis and policy rollback.
A high-quality sample answer
I would first confirm the synchronous budget and the high-risk categories. With an assumption of 278 requests per second average and 2,778 at peak, I would limit the synchronous path to hashes, format checks, lightweight models, and regional rules, targeting no more than 100 ms. Clearly harmful content is rejected immediately; the middle band enters a durable human queue; low-risk content publishes with a rescan marker. Deep video and audio analysis runs through queues and workers rather than blocking every user.
Thresholds are configured per policy category, not as one score for all risks. Each decision stores the content ID and version, model and policy versions, score, evidence summary, and action. Review tasks are ordered by expected exposure, risk, reports, and deadline, and leases prevent duplicate claims. Queue pressure protects high-risk work first. Reviewer decisions and reversals are immutable events, and appeals are handled by a different reviewer with the original evidence.
I would monitor pre-screen latency, queue age, false positives and misses by category, exposure time, appeal reversals, reviewer capacity, and unit cost. A model outage still runs hash and rule blocking; unknown results use restricted publishing or a human fallback. Model, policy, and regional changes are replayed offline and then canaried. The goal is an explainable, reversible system that continuously reduces harm, not a single offline accuracy number.
Common mistakes
- Run a heavy model synchronously for every item → Video and audio latency overwhelms the publishing path → Keep synchronous work to fast pre-screening and move deep analysis to queues.
- Use one threshold for every category → High-harm misses and contextual false positives cannot both be controlled → Configure thresholds by policy and region, with a human band.
- Allow content when a model returns no result → A timeout becomes an unaudited miss → Retain rule and hash blocking and route unknown results to restricted or human handling.
- Store only the final label → An appeal cannot reconstruct the decision → Store content, model, policy, score, evidence, and action versions.
- Serve the queue strictly FIFO → Popular or low-value traffic starves high-risk cases → Order by risk, expected exposure, and deadline while isolating capacity.
Follow-up questions
Follow-up 1: Is reviewing only 1% of total volume enough?
The percentage is not the goal. AWS documentation uses its image and video workflow as an example where machine filtering can leave roughly 1%–5% for human review; your system still validates category recall, harm of a miss, queue age, and exposure time rather than treating that range as a universal guarantee. If a high-risk category is missed, widen the human band or tighten publishing admission.
Follow-up 2: What if the review queue is backed up for a day?
Reorder by risk and expected exposure, pause low-value batch work, add high-risk review capacity, and raise alerts. Rescan and rate-limit already-published items; do not lower the backlog number by silently dropping tasks. If the high-risk deadline still cannot be met, tighten publishing and expose the capacity shortfall to the business.
Follow-up 3: How do you stop a successful appeal from being misclassified again?
Keep the policy and model versions from before and after the appeal, add reversals to an independent evaluation set by category, and check for regional, language, or group bias. Replay historical samples before changing thresholds or rules, then canary the change. An appeal result is not an unreviewed training label.
Follow-up 4: How do you preserve traceability when changing model vendors?
Define a normalized adapter output and version for each model, and retain a summary of the vendor result plus the mapping rule. Compare the new model on a fixed replay set and blind human review for category errors, misses, latency, and cost before regional or traffic canaries. On regression, roll back by policy version to the old adapter.