Prompt and use case
A global service already assigns tenants to multiple cells, yet one bad release, shared dependency failure, or capacity event still causes a broad outage. Audit the current architecture, identify paths that propagate impact across cells, and design repeatable fault-injection and recovery checks that prove each cell is a real isolation boundary.
What the interviewer is testing
- Whether you define a cell as a complete replica that can run, scale, and roll back independently.
- Whether you can reason about blast radius, routing maps, and consistent assignment of new users.
- Whether you handle shared control planes, cross-cell data, capacity skew, and hot-tenant migration.
- Whether you turn progressive rollout, metric gates, and recovery drills into concrete operations.
Questions to clarify before answering
- Is the isolation key a user, tenant, region, or business partition, and is cross-region access allowed?
- Which data must be globally consistent, and which data may be eventually consistent within a cell?
- Is the objective to limit deployment impact, survive a regional disaster, or both?
- Is load predictable, can a hot tenant move, and how is idempotency maintained during migration?
A 30-second answer framework
I would first map every cell's compute, cache, queue, data, and control-plane dependencies and mark every shared path. Then I would define isolation invariants for routing, capacity, releases, and cross-cell work. I would inject one-cell overload, a bad release, a data dependency outage, and control-plane loss, verifying that healthy-cell error rate and latency stay within gates. Any dependency that propagates impact must be partitioned, quota-limited, or backed by a local-snapshot fallback. Recovery time, affected-tenant percentage, and drill evidence determine whether isolation is real.
Step-by-step deep dive
1. The cell boundary
A cell is a system unit that can be deployed, scaled, and recovered independently, usually containing service instances, caches, queues, and a dedicated data shard. Shared DNS, identity, or configuration services should remain low-privilege and loosely coupled, with defined degradation behavior.
2. Routing and mapping
The routing layer looks up a map by user or tenant key and forwards the request to a target cell. The map needs a version, checksum, and expiry policy. During migration, write the new map first and drain old requests so one entity does not write to two cells at once.
3. Data isolation
Each cell owns a local write boundary; cross-cell queries prefer asynchronously replicated read projections. Global uniqueness belongs in a dedicated coordinator, or becomes partition-local uniqueness plus eventual reconciliation, instead of concentrating every write at one shared point.
4. Capacity and hot spots
Set request, queue, database-connection, and storage quotas per cell to bound the maximum blast radius. Monitor tenant distribution and resource watermarks. A hot tenant can move to an idle cell, but migration needs dual-read, single-write, and replayable events.
5. Shared control-plane risk
The control plane manages maps, versions, and orchestration; it should not carry all business data. During a control-plane outage, routers can serve from a local snapshot with a TTL. After expiry, allow only safe reads or return a retryable error.
6. Rollout and rollback
Run the new version in one cell and compare it with baseline cells on error rate, tail latency, resource saturation, and business metrics. Expand in batches after passing gates. Any regression pauses expansion and rolls back that cell without degrading healthy cells.
7. Cross-cell operations
Cross-cell reports, batch jobs, and account migrations run through an orchestrator with idempotency keys, leases, and progress checkpoints. On failure, retry only unfinished shards and expose partial completion explicitly to callers.
8. Disaster recovery and drills
Back up each cell’s data and configuration, with defined recovery-point and recovery-time objectives. Drill single-cell isolation, control-plane loss, regional failure, and mapping corruption to verify traffic transfer, data restore, and rollback scripts rather than relying on theoretical redundancy.
High-Quality Sample Answer
“I would not assume isolation merely because the service is deployed into several cells. I would trace a tenant request through databases, queues, caches, identity, configuration, routing, and the release pipeline to identify shared failure domains. I would then define three invariants: exhausting one cell cannot consume another cell's resources, a bad build can enter only one rollout batch, and the data plane can continue from a local routing snapshot during a short control-plane outage.
In a staging environment or a tightly isolated production drill, I would inject one-cell overload, database loss, bad configuration, and routing-directory loss. Healthy cells must remain within error-rate, P99 latency, and saturation gates. Any shared dependency that spreads failure gets partitioning, per-cell quotas, or a versioned local-snapshot fallback. Recovery checks cover traffic isolation, data integrity, and mapping rollback. I would make affected-tenant percentage, recovery time, and cross-cell health the release evidence, so the team proves isolation repeatedly instead of relying on the architecture diagram.”
Common Mistakes
- Replicating stateless compute while continuing to share a database, queue, or connection pool.
- Checking whether the failed cell recovers without checking healthy-cell error rate and tail latency.
- Using random routing instead of a stable tenant mapping, causing cross-cell writes.
- Testing infrastructure loss but omitting bad releases, hot tenants, and control-plane failure.
Follow-Up Questions and Responses
How do you know whether a shared control plane breaks isolation?
Disconnect it during a drill and verify that routers can serve from versioned local mappings with a TTL. After the snapshot expires, the system needs an explicit safe-degradation state.
What should an isolation acceptance gate measure?
Set limits for healthy-cell error rate, P99 latency, saturation, and affected-tenant percentage, and record recovery time. Recovery of the failed cell alone does not prove isolation.
Do more cells always mean better isolation?
No. Smaller cells reduce theoretical blast radius but increase capacity fragmentation, rollout batches, and cross-cell operational cost. Derive the count from the target affected-tenant percentage, cell capacity, and sustainable operating cost.
Can cross-cell reporting recreate a shared failure?
Yes. Reporting should read asynchronous projections with separate quotas and degradation. Online requests should not synchronously fan out to every cell.