Prompt and scope
A multi-tenant platform already stores audit events, but its security team needs an independent way to determine whether historical records were deleted, reordered, or recomputed by someone with primary-database access. Design the verification protocol, focusing on canonicalization, segmented hash chains, signatures, external anchors, verifier output, and the handling of unverifiable intervals.
Append-only records and hash chains make later edits detectable, but they do not prove an event was truthful when first written. The question tests threat modeling, evidence, and operations; immutable object storage alone is not a complete design.
What the interviewer evaluates
Evaluate event fields for actor, target, action, time, source, and result; reliable coupling to business transactions; segmented and anchored hash chains; tenant isolation, indexes, key permissions, retention, deletion, and a usable verifier.
Questions to Clarify Before Answering
- Can the threat actor modify the event store, hash catalog, signing keys, and anchor medium?
- Must verification detect deletion, modification, and reordering, or also attest to the event source?
- Will the platform, a security team, or an external auditor run the verifier?
- How long may records remain unanchored, and how should investigation workflows degrade after a gap?
30-second answer framework
“I would define an immutable canonical event and publish it through an outbox after the business transaction commits. An isolated writer groups events by tenant and time, builds hash chains, and periodically anchors each chain head in an independent append-only store or transparency ledger. Queries use rebuildable indexes, never mutate evidence. A verifier recomputes chains and compares anchors. Integrity comes from the chain; event truth still comes from business authorization and transaction records.”
Step-by-step deep answer
Step 1: Define the threat model and evidence goal
Assume an application or database administrator may delete, reorder, or rewrite records, and the anchor service may be unavailable. Decide whether the goal is to detect missing, changed, or reordered events, or to prove that a subject acted. An audit log does not replace access control.
Step 2: Design canonical events
Include event ID, tenant, actor and identity source, action, resource, before and after digests, request ID, event time, server sequence, result, and policy version. Canonicalize JSON and similar fields so serialization order does not change hashes. Store only digests or controlled references for sensitive values.
Step 3: Couple business and audit writes
The business transaction writes an outbox; a reliable consumer emits the event after commit. Do not rely on best-effort logging after a business write. Retries are idempotent by event ID; failures go to an isolated queue and alert. Audit latency may be bounded, but loss cannot be silent.
Step 4: Build segmented hash chains
Store the previous hash, canonical event, and current hash, such as H(previous || event || metadata). Roll segments by tenant, date, or size and record segment start, end, and sequence. Cross-shard order uses server sequence and receive time rather than client clocks alone.
Step 5: Anchor externally and manage keys
Periodically write each segment head to an isolated append-only medium with anchor time, segment ID, and signature. Keep signing keys in a controlled key service; rotation and revocation are themselves audited. An independent anchor prevents a database writer from recomputing and replacing an entire chain invisibly.
Step 6: Isolate queries and permissions
Build actor, resource, action, and time indexes in a search index or read replica. Indexes are rebuildable; evidence is not updateable. Enforce tenant, role, and purpose checks, and include a verification digest in exports. Every read, export, or verification is a new audit event.
Step 7: Handle retention, deletion, and privacy
Set retention, WORM, or object-lock policies by regulation. For personal-data deletion, use cryptographic erasure, field redaction, or irreversible digests while retaining proof that deletion occurred; never silently rewrite history. Align backup, cache, and anchor retention.
Step 8: Verify, recover, and monitor
The verifier checks chain continuity, sequence, canonical hashes, signatures, and external anchors. Run periodic sample and full verification, monitoring gaps, lag, duplicates, anchoring failures, and verification time. Recover by replaying outbox and append-only segments from the last trusted anchor, isolating unprovable ranges.
Deeper Trade-offs and Boundaries
#### One Chain Versus Segmented Chains
A single chain simplifies queries but complicates recovery and concurrent writes. Segments enable tenant isolation, parallel verification, and retention management, at the cost of segment anchors and ordering metadata.
#### Hash Chain Versus Signed Log
Hash chains are inexpensive and detect edits; signatures add cross-organization verification and key-management cost. Combine them when external proof is required.
#### Integrity Versus Truth
A chain proves relationships between records and anchor consistency, not that event content was true. Truth still depends on identity, authorization, transactions, and independent evidence.
Failure Drills and Evolution
#### A Middle Record Is Deleted
Delete one object from a segment and verify a sequence gap and chain break are reported with the segment and anchor location.
#### A Database Administrator Recomputes Hashes
Rewrite a segment and replace its head; the independent anchor must reject the new chain while the old chain remains retrievable for investigation.
#### The Anchor Service Is Unavailable
Disconnect it and verify segments enter a pending-anchor state, events remain durable, and anchors are added in order after recovery with an alert.
High-Quality Sample Answer
“I would first define the verification boundary: an attacker may alter the event store and search index but cannot also rewrite an independent anchor or obtain the verification key. The writer converts each event into a deterministic canonical representation, segments records by tenant and time, and stores a sequence number, previous hash, and current hash. It signs each segment end and periodically commits the chain head to an append-only medium under separate permissions.
An independent verifier starts from a trusted anchor, recomputes canonical hashes, and checks sequence continuity, links between segments, signatures, anchor times, and retention policy. Deletion creates a sequence gap, modification creates a hash mismatch, reordering breaks predecessor links, and an administrator who recomputes a whole segment still cannot match the external anchor. The result reports the verified range, first failing position, unanchored window, and evidence references rather than one Boolean. If an interval cannot be verified, I freeze affected exports, preserve original objects, and notify security. The chain establishes record integrity; identity, authorization, and business-transaction evidence are still required to establish event truth.”
Common Mistakes
- Putting data in WORM storage without defining who can write or change retention.
- Hashing raw JSON without canonicalizing field order, encoding, and timestamps.
- Treating a valid hash chain as proof that every event was truthful.
- Automatically rebuilding and replacing a failed chain, destroying investigation evidence.
Follow-Up Questions and Responses
What does an external anchor solve?
It prevents an operator with primary-store write access from silently recomputing a whole segment and replacing its chain head. The anchor needs separate permissions and retention.
How can the verifier prove a record was deleted?
It detects both a sequence gap and a broken subsequent hash, then uses the nearest trusted anchors to bound the missing interval. Absence from a search index alone is insufficient.
What happens while the anchor service is unavailable?
Continue writing sequenced local segments and mark them pending. Submit them in order after recovery. The unanchored window must appear in verifier output and alerts and cannot be described as fully verified.
How should an unverifiable interval be handled?
Quarantine affected exports, preserve original objects, signatures, and anchor evidence, notify security, and record the response. Recovery must not relabel an unknown interval as complete.