Representative interview topic

Data engineering interview: How would you design an auditable data-access governance layer?

DataHard
Offer.cc Editorial TeamPublished Updated

Question

A company has a lakehouse, warehouse, and several analytics engines. Analysts want self-service access, but sensitive columns must be constrained by purpose, role, and region, and the company must answer who read what and when. How would you design the governance layer?

Prompt and scope

Business data lives in object storage, lakehouse tables, and a warehouse; analysts use SQL, Spark, and BI tools. Access is maintained through manual tickets and local engine accounts. Offboarding is slow, and an audit cannot reconstruct cross-account queries. The goals are self-service requests, least privilege, row and column protection, emergency access, and one audit trail.

Design the governance layer: classification, approvers, policy decision and enforcement points, multi-engine coverage, break-glass, revocation, caches, exports, and proof that both authorization and actual reads are auditable. The core skill is data-platform governance and operations, so this is a data question.

What the interviewer evaluates

The interviewer wants a loop of catalog, policy, enforcement, evidence, and operations rather than “add RBAC.” Strong answers distinguish identity, purpose, resource labels, row/column filters, and masking, and explain the boundary between decision and enforcement.

They also address cross-service consistency. The same user reaching Athena, Spark, BI, or an export job should not lose authorization context or audit fields. Logs must be tamper-evident, searchable, retained deliberately, and include denials, administrative changes, and emergency access—not only successful queries.

Questions to clarify first

  • What data sources, engines, identity provider, and cross-account boundaries exist?
  • Which columns are personal, financial, or contract-restricted, and who maintains classification?
  • Is access decided by roles, attributes, purpose, tenant, row/column filters, or a combination?
  • Are approvals one-time, renewable, or required per query?
  • What are the maximum duration, approver, and review for emergency access?
  • Can the system record scanned scope, returned rows, export destination, and service identity?

A 30-second answer

“I would create an owned catalog and sensitivity labels, then send identity, team, purpose, region, and resource labels to a policy decision point. Enforcement at each engine or data proxy would apply row/column filters and masking, with time-bounded grants. Allow, deny, policy version, query resource, service identity, and export events would enter a centralized tamper-evident stream. Break-glass access would expire and trigger review. I would test revocation, cross-account context, caches, and policy drift to prove actual reads match the audit trail.”

Step-by-step solution

Build an asset inventory for tables, columns, paths, topics, tenants, and derived data. Each asset needs an owner, classification, source, retention, and approved purposes. Classification cannot be a one-time scan: schema changes, new columns, and business-rule changes trigger review. Uncertain sensitive fields should receive conservative labels until reviewed.

Separate decision from enforcement. The decision input includes subject identity, groups, attributes, purpose, device or network conditions, resource labels, and environment. The output includes allow or deny, filters, masking, policy version, and expiry. Enforcement may live in a proxy, engine plugin, or table filter, but bypassing the query engine must not expose raw object-storage files.

Self-service requests show purpose, fields, duration, and owner. Low-risk pre-approved uses can receive short-lived roles automatically; sensitive access needs the data owner or compliance approval. Grants expire quickly and are renewed deliberately. Offboarding, team changes, and project completion trigger revocation. Link approval evidence to the policy version that actually went live.

Explain row and column semantics. An analyst may see a masked email and aggregate result but must not reconstruct the original through joins, exports, or error messages. Tenant predicates come from trusted identity, not a user-supplied tenant ID. Apply equivalent controls to exports, temporary tables, caches, and materialized views so a protected source does not create an unprotected copy.

An audit event should contain subject, identity chain, purpose, resource, row/column decision, policy version, engine, query or job ID, time, source account, scanned or returned size, export destination, and allow or deny. Write to low-privilege append paths and controlled readers in immutable storage; use checksums or a version chain to detect deletion and tampering. Denials and policy edits matter as much as successful queries.

Across engines and accounts, preserve the original user and service identity. A service acting for a user keeps its delegation chain; cross-account events are copied to the resource owner. An old engine that cannot carry context is isolated, restricted to prefiltered views, or marked as an audit gap rather than declared compliant.

Design break-glass for a small, strongly authenticated group. Require a reason, short automatic expiry, immediate alerts, and review; emergency access never bypasses audit. Release policy versions with canaries and rollback. Monitor denials, privilege attempts, revocation latency, unowned assets, audit gaps, and high-risk exports. Synthetic identities and honey columns test whether data can be read and whether the read is recorded.

Model answer

“I would create an owned catalog and sensitivity labels covering source, derived, object paths, and exports. A policy decision point would receive identity, team, purpose, region, and resource labels and return allow, deny, row/column filters, masking, policy version, and expiry. Enforcement would sit in a proxy or engine plugin, while raw object paths remain inaccessible.

Requests would state purpose, scope, duration, and owner; sensitive access would need approval, and grants would expire on offboarding or project completion. The same row/column policy would apply to temporary tables, caches, and exports so users cannot reconstruct values through a copy.

Audit events would preserve user and service identity chains, resource, policy version, engine, query ID, filtering result, size, export destination, and allow/deny outcome in immutable storage. Break-glass access would expire, alert, and require review. I would test cross-account context, revocation, legacy engines, caches, exports, and honey columns to prove actual reads match the evidence.”

Common mistakes

  • Designing only RBAC → purpose, region, and column limits are missing → combine identity, attributes, labels, and filters.
  • Protecting only the query engine → users bypass it through object storage → secure the raw path and enforce one access boundary.
  • Logging only a shared ETL account → user accountability disappears → preserve the delegation chain.
  • Recording only successful queries → denials and drift disappear → record denials, policy edits, and emergency use.
  • Permanent grants → project access survives forever → short expiry, renewal, and revocation.
  • Protecting only source tables → caches, exports, and materializations leak → enforce the policy on every derived path.
  • Break-glass bypassing audit → the emergency path becomes a backdoor → strong auth, reason, expiry, alert, review.
  • Testing policy output only → filters can be bypassed → test real reads, exports, synthetic identities, and honey columns.

Follow-up questions and responses

Follow-up 1: RBAC or ABAC?

Use roles for stable team boundaries and attributes for dynamic purpose, region, labels, and time. Most real systems combine them while limiting policy complexity.

Follow-up 2: How do you stop reconstruction through aggregates?

Limit small cohorts, joins, differencing queries, and export frequency; use thresholds or noise where appropriate. Validate rules against sensitivity and accuracy requirements.

Follow-up 3: How do you audit cache reads?

Bind cache keys to subject and policy version, record fill and read identity chains, and invalidate on revocation or policy change. Shared caches without user context must not hold sensitive results.

Follow-up 4: What if the audit log contains sensitive SQL?

Store structured resource identifiers and safe summaries instead of full SQL, parameters, or raw data. Encrypt, restrict, and retain audit fields deliberately.

Follow-up 5: What if a legacy engine cannot enforce row or column policy?

Isolate it, expose only prefiltered views, or migrate it behind a proxy, and mark the remaining audit gap. Team convention is not enforcement.

Follow-up 6: How do you prove governance does not slow analysis?

Track decision latency, query p95, false denials, cache hit rate, and approval time. Invalidate cached decisions immediately on identity or policy-version changes.

Follow-up 7: Why record policy versions?

The same query can return different results before and after a policy update. A version explains why access was allowed, denied, or filtered and supports rollback and incident review.

Public sources

Related questions