Prompt and context
Kubernetes v1.36 promoted VolumeGroupSnapshot, VolumeGroupSnapshotContent, and VolumeGroupSnapshotClass to groupsnapshot.storage.k8s.io/v1. The interviewer wants to know how several PersistentVolumeClaims form one recovery point, what the CSI driver owns, and how you handle the gap between storage-level and application-level consistency.
What the interviewer evaluates
- Whether you distinguish crash consistency, application consistency, and eventual consistency.
- Whether you can explain the boundary between Kubernetes controllers, the external snapshot controller, and the CSI driver.
- Whether you identify prerequisites such as CSI support, capacity, topology, and snapshot lifecycle limits.
- Whether you turn recovery objectives, drill metrics, and failure rollback into an executable process.
Questions to clarify first
Confirm whether the database supports online backup, whether every volume uses the same CSI driver, and the required RPO, RTO, and cross-zone behavior. Also confirm whether the storage system preserves write ordering or whether the application must flush, freeze, or pause writes first.
A 30-second answer
I would use four layers: the application creates a consistency point; Kubernetes records the PVC set with VolumeGroupSnapshot; the CSI driver creates a group snapshot in storage; and recovery restores new volumes and validates them. The API is GA in v1.36, but it coordinates crash-consistent volume snapshots; it does not replace database logs, key management, or recurring recovery drills.
Step-by-step deep dive
1. Establish a consistency point
The database performs a provable backup action, such as briefly freezing writes, flushing WAL, or creating a checkpoint. Record the transaction position, snapshot time, and application version before creating the group snapshot. A storage-only snapshot without application coordination can be disk-consistent while the business state is inconsistent.
2. Create and observe the group snapshot
Create a VolumeGroupSnapshot for the PVCs of one database instance and wait for its ready status. Controllers coordinate Kubernetes objects; the CSI driver performs storage operations and must support the volume-group snapshot extension API. Record each member's snapshot handle, capacity, topology, and error so partial success is never mistaken for recoverability.
3. Recover and validate
Create new PVCs for every member while preserving the database-to-volume mapping. Start the database in an isolated environment, validate the WAL position, table counts, checksums, and key business queries, then switch traffic. Cross-zone recovery must also verify that the snapshot replica is readable and that the CSI driver accepts the target topology.
4. Operational boundaries
Define retention, encryption, and access controls; monitor snapshot duration and failure rate. Treat snapshots as recovery material rather than permanent archives: export them to independent media and measure actual RPO and RTO through drills. Before deleting PVCs, snapshot objects, or backend objects, confirm the reclaim policy will not remove a recovery point still in use.
Example of a strong answer
I would define recovery objectives first and select PVCs managed by one CSI driver. The application creates a checkpoint and records its WAL position, briefly pausing writes when needed; then we create a groupsnapshot.storage.k8s.io/v1 VolumeGroupSnapshot. The controller coordinates Kubernetes objects, while the CSI driver and storage system provide the group semantics. I would verify driver version, topology, capacity, and snapshot quotas, and alert on every member's ready state.
Recovery never overwrites production volumes. It creates new PVCs from the group snapshot, starts the database in an isolated namespace, validates WAL, checksums, and business queries, and only then switches traffic. If any member fails, the group is unusable and the workflow rolls back; partial snapshots are not called a complete backup. Scheduled drills prove RPO and RTO, while independent archival, encryption, and least privilege prevent the snapshot system from becoming a single point of failure or leakage.
Common mistakes
- Assuming GA means every storage driver supports the feature automatically; it still depends on CSI and storage implementation.
- Describing creation only, without member-level failure, topology, and lifecycle handling.
- Calling crash consistency application consistency and omitting flush, WAL, or checkpoints.
- Keeping only snapshot metadata and never starting the recovered database in an isolated environment.
Follow-up questions and responses
What if one member volume fails to snapshot?
Mark the group snapshot unusable, retain events and created handles, clean orphaned resources, and retry. Recovery accepts only a complete member set, while partial success is reported for alerting and capacity audit.
Why not snapshot each PVC separately?
Individual snapshots lack a shared recovery point, so cross-volume write order can diverge. A group snapshot delegates the member set and group operation to storage, while application checkpoints are still required for business consistency.
How do you prove the plan meets RPO and RTO?
Recover in an isolated cluster on a fixed schedule, recording time from checkpoint to queryable service, data-position drift, and failure rate. Drill results become release gates; if thresholds are missed, adjust snapshot frequency, archive paths, or cutover procedures.