Prompt and scope
Design a multi-tenant audit log service that records security- and compliance-relevant actions: sign-ins, permission changes, data access, and configuration updates. Investigators need searchable history and verifiable exports, while application traffic must remain available during a logging outage.
Define the integrity promise precisely. AWS CloudTrail describes an immutable, searchable event history, while OpenTelemetry provides a common structured log model; neither claim that an event was correct before it was emitted. Your design should state what the service can prove.
What the interviewer is testing
They are testing durable ingestion, append-only storage, integrity verification, tenant isolation, retention policy, and operational trade-offs. A good answer separates audit evidence from ordinary debug logs and explains how to avoid losing or mutating security events during overload.
Questions to clarify before answering
- Which actions are mandatory, and what is the expected events-per-second and burst size?
- Is the requirement tamper-evident, tamper-resistant, or externally attestable?
- Must the application request fail if an audit event cannot be accepted?
- How long must each tenant retain events, and do legal holds override deletion?
- Who can search, export, or verify another tenant’s data?
- What query dimensions and export formats do investigators need?
A 30-second answer framework
“I would expose a regional ingestion API and a local durable buffer so application calls do not depend on the search index. Each event carries tenant, actor, action, target, request correlation, event time, ingest time, schema version, and source. Partition an append-only log by tenant and time, replicate it, and create a hash chain or signed segment manifest so later edits are detectable. Separate hot searchable indexes from immutable retention storage. Enforce tenant-scoped authorization, retention and legal holds, and provide an export with verification metadata. Monitor accepted-versus-dropped events, ingestion lag, integrity checks, and export completion.”
Step-by-step deep dive
Step 1: Define the event contract
Require an event ID, tenant ID, actor and authentication context, action, target, outcome, source service, request ID, event time, ingest time, schema version, and selected attributes. Keep secrets and unnecessary payloads out of the record; record a reference or a redacted summary instead.
Step 2: Separate acceptance from indexing
Return success only after the event reaches a durable buffer or replicated log. Consumers then build search indexes and exports asynchronously. This prevents a search-cluster incident from silently deleting evidence or making every application request wait on indexing.
Step 3: Make integrity verifiable
Canonicalize each event, hash it with the prior event or segment root, and periodically sign or anchor the manifest in a separate trust domain. Store sequence gaps and verification results. A hash chain detects changes after ingestion; it does not prove that an upstream service emitted a truthful event.
append(event):
canonical = canonicalize(event)
record.hash = H(previous_hash || canonical)
durable_log.append(record)
return accepted(record.event_id, record.hash)Step 4: Partition and replicate
Partition by tenant and time, with a hash or tenant key to distribute hot tenants. Replicate across failure domains before acknowledging according to the durability target. Preserve ordering per tenant or aggregate, but do not promise global ordering unless the cost is justified.
Step 5: Build hot search and cold retention
Index recent events for investigator queries and compact older segments into immutable object storage. Keep the index rebuildable from the log. Retention jobs must respect per-tenant policy and legal holds; deletion should leave an auditable policy record without retaining the protected payload.
Step 6: Enforce access and export controls
Authorize every query by tenant, role, purpose, and time range. Log reads and exports as audit events too. Produce a signed manifest containing filters, event counts, segment hashes, and timestamps so a recipient can verify completeness and detect alterations.
Step 7: Define outage and overload behavior
Choose a bounded local buffer, backpressure, and a clear policy for mandatory events. For non-critical telemetry, sampling or delayed delivery may be acceptable; for security events, reject the originating mutation or route to an isolated emergency channel. Never report accepted when the event was only held in volatile memory.
Step 8: Operate the trust boundary
Measure ingestion lag, durable-accept latency, consumer lag, rejected events, sequence gaps, hash verification failures, index freshness, export duration, and retention-job errors. Restrict key access, rotate signing keys, test restore and verification, and alert on an audit trail of the audit trail.
High-Quality Sample Answer
“I would treat the audit log as an independent evidence system. A business service emits a canonical event with an event ID, tenant, actor, authentication context, action, target, result, request ID, and schema version. The service acknowledges receipt only after the event reaches a replicated durable log. Asynchronous consumers build search indexes and exports, so a lost index can be rebuilt from evidence.
The evidence log is segmented by tenant and time, links records with hashes, and periodically signs or anchors segment heads. Queries are authorized by tenant, role, purpose, and time range; queries and exports are themselves audited. An export includes its filters, event count, segment hashes, and a signed manifest. During ingestion failure, security events enter a bounded durable emergency path or the original high-risk action is rejected; an in-memory buffer is not an accepted event. Operations monitor ingestion latency, sequence gaps, hash failures, index freshness, rejected events, and retention errors, and regularly drill index rebuilds, key rotation, and hot-tenant isolation. This service proves integrity after receipt; identity, authorization, and business transactions still establish whether the upstream event was truthful.”
Common Mistakes
- Acknowledging an event before durable storage, creating silent gaps after a process crash.
- Treating a mutable database table or ordinary debug log as audit evidence.
- Making the search index the only copy, so corruption cannot be rebuilt.
- Forgetting that queries, exports, retention changes, and key rotation also require auditing.
- Claiming a hash chain proves that an upstream event was truthful.
Follow-Up Questions and Responses
What happens if the entire search index is lost?
Keep evidence ingestion available, rebuild the index from immutable segments, and verify the rebuilt range with event counts, sequences, and signed manifests.
Should business traffic continue during an ingestion outage?
Decide by event criticality. Low-risk actions may enter a bounded durable buffer. A high-risk security action that cannot be recorded reliably should be rejected or routed through an isolated emergency path, never silently allowed.
How do you stop one hot tenant from harming others?
Partition by tenant and time, enforce ingestion, query, and export quotas, and isolate consumer resources. Load tests must verify durability and query service-level objectives for unaffected tenants.
How do privacy deletion and audit retention coexist?
Classify fields by legal hold and tenant policy, prefer redacted digests or controlled references, apply deletion consistently to indexes, object segments, and backups, and retain a verifiable record of the policy action.