1. Question
A platform receives time-series metrics labeled by service, tenant, and region. Short-term debugging needs second-level data, while long-term reports need minute or hour trends. Ingestion keeps growing. Keep 15 days of raw samples and 2 years of aggregates while controlling cost and preventing high cardinality from overwhelming queries.
2. Constraints and clarifications
- Confirm scrape interval, label count and cardinality, query ranges, SLO, acceptable data delay, and deletion requirements.
- Separate raw samples, fixed-window aggregates, recording-rule results, and long-term archives.
- Do not replace counters, histograms, or quantiles with a simple average; aggregation must preserve metric semantics.
- Define at-least-once ingestion, duplicate samples, out-of-order timestamps, and tenant isolation.
3. Core architecture
The ingestion layer applies label allowlists, cardinality budgets, and batch compression before writing time-partitioned hot storage. The query layer routes by range and step: short ranges read raw blocks, long ranges read pre-aggregated blocks, and mixed ranges are merged with resolution metadata. A replayable window job reads raw data, writes versioned aggregate blocks, verifies them, and only then deletes expired raw blocks.
4. Reference flow
ingest(sample):
series = canonicalize(metric_name, sorted_labels)
enforce_cardinality_budget(series)
append_to_time_partition(series, sample)
downsample(window):
raw = read_raw(window)
agg = aggregate_by_metric_semantics(raw, resolution=5m)
write_versioned_block(window, agg, source_watermark)
verify_counts_checksums_and_watermark(agg)
query(range, step):
blocks = choose_resolution(range, step)
return merge_with_gap_and_resolution_metadata(blocks)For counters, preserve increments and resets; for gauges, preserve min/max/average; for histograms, merge buckets or native histogram structures. Return actual resolution, coverage, and gaps so users do not mistake downsampled data for raw precision.
5. Consistency and cost trade-offs
Late samples or repeated execution can change an aggregate, so jobs need watermarks, versions, and idempotent writes, plus a defined correction period after window close. High-cardinality labels multiply memory, index, and query cost; limit arbitrary user labels with tenant quotas or pre-aggregation. Higher resolution, longer retention, and lower query latency have direct storage and compute costs.
6. Verification and observability
- Validate input/output sample counts, counter increments, bucket totals, checksums, and watermarks for every window.
- Replay the same window repeatedly and confirm idempotency and replaceable result versions.
- Monitor ingestion rejection rate, series count, query samples, downsampling delay, gap rate, and storage cost.
- Compare raw and aggregate results for real incidents to ensure spikes, resets, and anomalies are not averaged away.
7. Common mistakes
- Averaging every metric by time and breaking counter, quantile, or histogram semantics.
- Using unsorted label strings as a series key, creating duplicate series and incorrect cardinality.
- Deleting raw blocks before downsampling is verified, making replay or late-data correction impossible.
- Routing only by time range and ignoring step, gaps, and resolution metadata.
8. Interview scoring points
Defines a tiered data model
The candidate separates raw, pre-aggregated, and archived data and states each tier's resolution, retention, and query purpose.
Preserves metric semantics
The candidate treats counters, gauges, histograms, and quantiles differently instead of applying one average function to all types.
Controls cardinality and cost
The candidate proposes label rules, tenant quotas, ingestion rejection, and storage budgets, then explains the query-cost relationship.
Designs replayable validation
The candidate uses watermarks, versions, idempotent writes, and replay checks while monitoring gaps, delay, and anomaly fidelity.