System design interview: Build an auditable, rollback-safe model registry

Prompt and context

Your platform serves hundreds of models whose training artifacts come from different pipelines. Online services need a stable reference to an approved version, while research wants fast experiments and compliance requires training-data snapshots, evaluation metrics, approvers, and deployment history. Assume model files live in object storage and inference services resolve versions and aliases through the registry.

What the interviewer evaluates

The interviewer wants you to separate immutable model versions from mutable release pointers. A strong answer covers metadata consistency, artifact integrity, atomic alias changes, environment isolation, approvals, and rollback. A weak answer builds only a CRUD service that stores filenames.

Clarifications to ask first

• Can a model be reused across environments? If development, staging, and production are isolated, aliases and permissions need separate boundaries.
• Is rollback to the previous, a specified, or the last healthy version? Each target changes the health and audit model.
• Which evaluation evidence is mandatory? A version missing a dataset snapshot or metric cannot be approved.
• Can inference services cache models? Cache TTL and alias propagation determine rollback time.
• Are canaries or multiple live versions required? If so, traffic routing must be tied to model-version metrics.

30-second answer framework

“I would make model files and version metadata immutable, address artifacts by content digest and version, and use one environment-scoped alias for the approved version. Registration, evaluation, approval, deployment, and rollback are audited state transitions. Inference resolves an alias to a version snapshot, then verifies digest and authorization. Releases use conditional updates, and a guardrail breach atomically moves the alias back to the last healthy version.”

Step-by-step deep answer

1. Define objects. RegisteredModel stores owner and policy. ModelVersion stores artifact URI, digest, framework, input/output signature, training-data snapshot, and evaluation metrics. Alias is a mutable environment-scoped pointer.
2. Make artifacts immutable. Compute a digest at upload and write the artifact to object storage. A version only gains metadata; its file cannot be overwritten. Verify the digest on download.
3. Use a state machine. Versions move through registered, validated, approved, deployed, and retired. Every transition records actor, reason, time, and evidence link; invalid transitions are rejected.
4. Switch aliases safely. Aliases such as candidate, staging, and champion point to one version at a time. Use conditional writes or a transaction so readers never observe a partial switch.
5. Resolve during deployment. The service resolves an alias to a version snapshot, checks tenant, environment, and digest, then downloads and caches the model. The cache stores the version, not only the alias.
6. Canary and rollback. A controller binds traffic groups to aliases or routing rules and collects error, latency, and business metrics. A breach moves the alias to a healthy version and records an automatic rollback event.

Store metadata in a relational database and large files in object storage. An event stream can invalidate caches and index audit events asynchronously. Cross-region replication should copy approved versions only, so an unfinished artifact cannot reach production.

Model answer

“I would enforce three boundaries: immutable versions, mutable aliases, and audited state transitions. Each version contains artifact digest, signature, training-data snapshot, evaluation metrics, and the source run. Approval is rejected when required evidence is missing. The production champion alias changes through a conditional update to exactly one version; inference resolves and caches the version number, then verifies its digest. A canary controller ties the model version to error, latency, and business metrics and atomically returns the alias to the last healthy version when a guardrail fails. Permissions are split by model, environment, and action, and alias moves, approvals, downloads, and rollbacks are written to tamper-evident audit logs.”

Common mistakes

• Mistake: Overwrite the file for an existing version → Why it fails: Deployments cannot be reproduced and audits point to changing content → Fix: Make versions and artifacts immutable.
• Mistake: Deploy the literal latest string → Why it fails: Alias moves make caches and audits inconsistent → Fix: Resolve to a version snapshot and record its digest.
• Mistake: Store approval as a Boolean → Why it fails: Nobody can explain who approved what and why → Fix: Use evidence-backed state transitions and audit events.
• Mistake: Roll back only traffic → Why it fails: Caches or dependencies may still load the new version → Fix: Version caches, check dependencies, and atomically move the alias back.

Follow-up questions and responses

Why use aliases instead of deploying version numbers directly?

Version numbers provide reproducibility; aliases provide release control. Separating them allows a switch without changing client configuration while preserving an explicit version audit trail.

How do you prevent concurrent alias updates from overwriting each other?

Store a version or revision number with the alias and use a conditional update. On conflict, reread policy and retry; never silently overwrite another release.

What does the registry store when training data is sensitive?

Store the dataset version, access policy, and digest, not raw samples. Reuse data permissions when fetching evaluation evidence and record the access audit.

When can old model versions be deleted?

Delete artifacts only after the retention period, active deployments, and audit holds are clear. Keep metadata and rollback digests through the compliance retention period.