System Design Interview: Design a Multi-Tenant Feature Entitlement Service

Prompt and context

This system-design prompt fits platform, backend, and SaaS infrastructure roles. A billing system emits plan and payment events; product services need an authorization decision such as can tenant T use feature F for subject U?. Assume 50,000 tenants, 10 million subjects, 100,000 decision requests per second at peak, and a 99.99% monthly availability target. A cancelled or suspended entitlement must not remain usable indefinitely, while a billing outage must not take down every read path.

What the interviewer is assessing

• Can you separate billing as the source of commercial truth from an evaluated entitlement snapshot?
• Do you define tenant and subject isolation, precedence between plan, add-on, trial, and deny rules?
• Can you reason about event ordering, cache staleness, revocation latency, and fail-open versus fail-closed decisions?
• Do you provide a versioned API, audit trail, observability, and a replayable recovery path?

Clarifying questions to ask

Ask whether decisions are per tenant, user, service account, or seat; whether a feature can be enabled for a subset; how quickly cancellation must revoke access; whether usage quotas are part of the decision; and whether every decision needs an explainable reason. Confirm whether billing events are at-least-once and can arrive out of order. These answers change the snapshot schema, event handling, cache TTL, and failover policy.

30-second answer framework

I would keep billing as the authoritative source and build an entitlement projection keyed by tenant, subject scope, feature, and version. A write path consumes ordered or deduplicated subscription events, computes a new snapshot, and publishes an invalidation. A read API evaluates the snapshot with explicit precedence and returns allow, deny, reason, and version. Regional caches serve hot reads, but a revocation token or version fence bounds stale access. Fail-open is allowed only for low-risk features; paid or security-sensitive features fail closed and expose a recovery path. Every change and decision is auditable.

Step-by-step deep dive

1. Define the contract. Evaluate(tenant_id, subject_id, feature, context) returns a decision, reason code, snapshot version, and expiry. Context may include plan, region, seat, or rollout attributes; OpenFeature requires a unique targeting key and supports custom fields, so do not overload a single free-form string with billing state.
2. Model immutable grants. Store subscription products, add-ons, trials, seats, effective and expiry times, and explicit denies as versioned facts. The projection stores the resolved feature set plus source fact IDs. A deny from suspension or compliance overrides a normal grant; a time-bound trial expires without mutating history.
3. Build the propagation path. Billing emits an event with tenant, subscription version, event ID, and effective time. An inbox deduplicates event IDs, rejects an older version, and writes the fact and projection transactionally. An outbox publishes entitlement_version_changed; consumers invalidate by tenant and feature. Replaying facts reconstructs a projection after corruption.
4. Serve reads. A stateless evaluation API reads a local cache or regional store. Cache keys include tenant, subject scope, feature, and policy version. Cache entries carry the projection version and expiry. If a request presents a newer version fence than the cache, read the authoritative regional store before deciding.
5. Choose consistency by risk. Set a measured revocation SLO, such as 60 seconds for ordinary cancellation and near-immediate fencing for fraud or security suspension. Publish a deny fence to a strongly reachable store; services reject cached allows older than that fence. Do not promise zero stale reads without paying the cost of synchronous checks.
6. Handle failures and scale. Partition events by tenant to preserve per-tenant order, shard projections by tenant hash, and keep hot tenants isolated. On billing lag, expose last-applied version and alert. On cache or regional-store failure, use a bounded stale window only for low-risk features; return a typed dependency error for high-risk features instead of silently granting access.
7. Audit and verify. Record who changed a plan, which event version produced a projection, and why a decision was made. Measure event lag, projection age, cache hit rate, stale-allow blocks, decision latency, and cross-tenant authorization failures. Test out-of-order events, duplicate delivery, clock skew, cancellation during a request, tenant migration, and replay from an empty projection.

High-quality sample answer

I would separate commercial truth from a versioned entitlement projection. Billing events carry an event ID, tenant, subscription version, effective time, and changed products. An inbox deduplicates and rejects older versions, then transactionally writes facts, the resolved feature snapshot, and an outbox notification. The evaluation API returns allow or deny, reason, projection version, and expiry. A cache key includes tenant and subject scope so one customer cannot read another customer’s decision.

The key trade-off is revocation. I would set a 60-second ordinary cancellation SLO and publish a deny fence for fraud or security suspension. Every cached allow carries a projection version; a newer fence forces a regional-store read. Low-risk UI features may use a bounded stale window during a store outage, while paid data export or security controls fail closed with a typed dependency error. Audit records link each decision to the event and policy version, and a replay job rebuilds projections from immutable facts.

Common mistakes

• Reading billing tables synchronously for every request → payment latency and outages become authorization outages → project immutable facts into a read-optimized snapshot.
• Caching only by feature → one tenant or subject can receive another scope’s decision → include tenant, subject scope, and policy version in the key.
• Applying events in arrival order → an old cancellation or renewal can overwrite newer state → deduplicate IDs and reject versions older than the applied version.
• Promising instant revocation everywhere → the design hides network and cache costs → state a measurable revocation SLO and enforce a deny fence.
• Failing open for paid or security features → stale access becomes a revenue or safety incident → classify features by risk and fail closed where required.

Follow-up questions and answers

How do you support a feature for only 10 percent of a tenant’s users?

Keep commercial entitlement and rollout targeting separate. The entitlement snapshot says the tenant owns the feature; an evaluation context with a stable subject targeting key applies the rollout rule. Record both decisions so a support engineer can distinguish “not purchased” from “not selected by rollout.”

What happens when a cancellation event is delayed?

Expose projection age and event lag, alert before the revocation SLO is breached, and use the billing version or deny fence when available. Do not infer cancellation from a missing heartbeat. Once the event arrives, apply it idempotently and invalidate all affected scopes.

How do you migrate a tenant between shards?

Write a migration epoch into the tenant metadata, dual-read during the bounded cutover, and publish a fence that prevents an older shard from serving allows. Verify counts, versions, and sampled decisions before removing the old copy; retain replayable facts for rollback.