Prompt and context
Build a multi-tenant WORM object service. Objects are versioned, can have a fixed retention period, and can be protected indefinitely by a legal hold. Customers require deletion and overwrite rejection during retention, governance and compliance modes, and audit paths that cannot be bypassed by administrators, replication, or lifecycle cleanup. Explain write, read, delete, replication, recovery, and audit flows.
What the interviewer is testing
- Whether you distinguish object versions, retention periods, governance mode, compliance mode, and legal holds.
- Whether immutability is enforced by storage execution rather than client convention.
- Whether you design permissions, governance bypass, root-account boundaries, and dual approval.
- Whether you handle replication, retries, delete markers, and clock consistency.
- Whether every policy change and rejection is provably auditable.
Questions to clarify first
- Are policies defined by bucket, prefix, object version, or tenant contract?
- Must compliance mode prevent deletion by the root account before expiry?
- Who starts, removes, and approves legal holds, and is four-eyes approval required?
- Must replicas preserve lock state and dates, and what cross-region lag is acceptable?
- Do clients need strongly consistent reads, version listing, proof export, or only delete rejection?
A 30-second answer
“I would separate object data, version indexes, and retention policy while committing their state atomically. A write creates an immutable version; the policy engine computes retainUntil, mode, and legal-hold state, and every delete path checks those facts inside the same authorization boundary. Governance mode permits an explicitly authorized bypass, while compliance mode rejects every delete before expiry; a legal hold has no date and can only be removed through an approval workflow. Replication carries version and lock metadata. Writes, rejected deletes, bypasses, and policy changes go to an append-only audit log that can produce evidence.”
Step-by-step deep dive
1. Model versions and lock state
An object key is not the protected entity; objectVersionId is. Store content digest, write time, retainUntil, mode, legal hold, tenant, and policy version. A simple delete creates a delete marker rather than physically deleting a protected version; permanent deletion must name a version and pass the lock check.
2. Design writes and default retention
Bucket or tenant policy can provide default mode and duration, while a write may request an object-level value. Validate maximum duration, time source, and permission, then atomically bind the computed metadata to the version. Policy changes affect future versions and cannot shorten existing retention retroactively.
{
"objectVersionId": "v_91c2",
"retention": {"mode": "COMPLIANCE", "retainUntil": "2027-01-01T00:00:00Z"},
"legalHold": "OFF",
"policyVersion": 18
}3. Enforce delete and overwrite gates
Delete, overwrite, shortening retention, and removing a legal hold use one authorization service. It reads the latest version and conditionally checks time, mode, principal permission, and hold state. Governance mode requires an explicit bypass permission and request marker; compliance mode rejects all deletes before expiry, including administrators. Rejections return a stable error and audit ID.
4. Design legal holds and approval
A legal hold is independent of retention and protects a version until explicitly removed. The removal workflow binds a case or contract, reason, actor, approver, and step-up authentication; high-risk tenants can require two people. After removal, an unexpired retainUntil still protects the version, so a hold is not a retention override.
5. Handle replication, recovery, and clocks
Replication transfers content, version ID, digest, and complete lock metadata; the target verifies the source signature and policy version before committing. The source keeps rejecting deletes until the target confirms them, and the target cannot claim compliance before lock verification. Recovery creates a new version and never mutates the old one. Compare dates using a controlled time service and monotonic audit timestamps, not one machine clock.
6. Build audit and proof
Audit writes, version creation, successful or rejected deletes, bypasses, legal-hold changes, replication confirmation, and policy publication. Store events in a separate append-only log with a hash or signature chain and restricted query access. Periodically export version inventory, lock state, and audit digests so a customer can prove protection at a point in time.
Example of a strong answer
“I treat version and lock metadata as immutable facts, and every delete path reads current state and performs a conditional check. On write, the policy engine computes retention, mode, and legal hold and atomically binds them to the version. Governance mode allows an explicitly authorized bypass; compliance mode rejects all deletion before expiry. A legal hold is removed only with a case, reason, and approval. Replication carries version, digest, and lock metadata and becomes compliant only after target confirmation. Rejections, bypasses, policy changes, and replication enter an append-only signed audit log, from which clients can export version inventory and proof digests.”
Common mistakes
- Checking locks only at the API layer → cleanup or replication paths bypass them → recheck in storage deletion execution.
- Applying a bucket-policy change to old versions → existing retention can be shortened illegally → version policies and affect future writes only.
- Treating a legal hold as a fixed date → automatic deletion after expiry may violate a case → keep holds independent with explicit removal.
- Making governance bypass implicit → privileged misuse cannot be explained → require explicit permission, marker, and approval audit.
- Declaring compliance when replication is submitted → the target may lose lock metadata → confirm target version and lock state first.
Follow-ups and responses
Why can a simple DELETE succeed without deleting a version?
With versioned objects, a simple DELETE can create a new delete marker that hides the current version while the protected version remains. Permanent deletion must specify the version ID and pass the lock gate.
Why does compliance mode need a stronger account boundary?
Its meaning is that no principal, including a root account or storage administrator, can delete before the retention date. Deletion must be isolated from ordinary IAM superuser grants so a privileged role cannot invalidate compliance.
How do you prove a rejected delete was not lost from audit?
Use one request ID for authorization decision, version state, and audit event, then write the event to a separate append-only log. Reconcile delete requests, version inventory, and rejection events periodically and freeze high-risk operations on gaps.
May the source be deleted while replication is delayed?
No. The source lock remains until the target confirms content digest, version ID, and retention metadata; otherwise a replication failure creates a window with no compliant copy.