Prompt and applicable context
Your company runs Java, Go, JavaScript, and PHP services. Today they rely on environment variables and language-specific SDK startup flags. The platform team wants OpenTelemetry declarative configuration files selected with OTEL_CONFIG_FILE, gradually standardizing resource attributes, sampling, exporters, and processors. Design the migration, compatibility, validation, and rollback plan.
In March 2026 OpenTelemetry announced stable portions of the declarative configuration JSON schema, file YAML representation, in-memory model, plugin-component mechanism, parse/create operations, and OTEL_CONFIG_FILE; language implementations still have different maturity. The interview tests cross-version configuration governance, not YAML memorization.
What the interviewer evaluates
- Whether you distinguish a stable specification from equally stable implementations in every language.
- Whether you can define a compatibility matrix for the schema, SDK versions, exporter endpoints, and permissions.
- Whether you can prevent old environment variables and new files from causing implicit overrides or duplicate exporters.
- Whether you can prove migration completeness without losing traces, creating high-cardinality data, or multiplying cost.
A strong answer treats configuration as a versioned artifact with linting, synthetic tests, canaries, comparison, and fast rollback. A weak answer only says “put everything in YAML.”
Clarifying questions before answering
- Which languages support the target fields, and which must keep environment variables? This determines migration batches.
- Is the file baked into an image, mounted as a volume, or downloaded at runtime? The source changes signing, permissions, and rollback speed.
- Does application code read existing environment variables? If so, deletion is unsafe until precedence and a deprecation window exist.
- Is temporary dual writing allowed during migration? It costs more but creates comparison evidence.
30-second answer framework
“I would inventory each language’s implementation status and configuration sources, then define a versioned minimum schema and compatibility matrix. A generator would render language-neutral YAML, while CI checks schema, permissions, endpoints, and cardinality; unsupported languages keep the old environment-variable path. I would validate with a no-business-traffic synthetic service, then canary by language and risk, comparing sent, dropped, sampled, latency, and cost metrics. Every artifact would carry a version and rollback pointer. If parsing fails, the launcher keeps the previous artifact or old flags instead of silently starting with no telemetry.”
Step-by-step deep answer
1. Build a capability matrix and boundary
Split configuration into resources, sampling, receivers, processors, exporters, and plugin components. For each language record SDK version, supported fields, defaults, environment-variable mapping, and unknown-field behavior. A stable data model and parse/create mechanism do not imply equally mature language implementations.
2. Define one source and precedence
Make the declarative file the primary source for new services while legacy services keep environment variables. During migration define precedence explicitly: an explicit file field overrides a platform-generated default; an unsupported field is rejected or marked by the adapter rather than silently ignored. Emit a redacted effective-configuration snapshot for audit.
repo template -> rendered config -> schema validation -> signed artifact
| |
env defaults --------------------------> runtime loaderDo not let every service assemble its own YAML. Platform templates own common settings; a service may declare a small, reviewed override.
3. Design the artifact and release path
The artifact needs a version, target language, SDK range, exporter endpoint, secret references, and compatibility declaration. CI validates the schema, then starts a minimal service to load it and establish exporter connections. Production uses an immutable digest and signature; rollback switches to a previously verified digest.
4. Handle old variables and duplicate exporters
Before migration collect environment injection, sidecars, and in-code configuration. If file and variables coexist, test precedence in staging and prohibit two exporters for the same signal. Keep old variables as an explicit fallback with a deprecation date and alert; otherwise two configuration systems will coexist indefinitely.
5. Verify semantics with synthetic telemetry
Generate fixed trace, metric, and log samples for each language. Check service name, resource attributes, span names, sampling, batch size, and exporter target. Do not stop at “the process started”; compare input and output counts, latency, rejection reasons, and high-cardinality labels in the Collector or backend.
6. Canary in batches and define stop conditions
Choose one business flow, one language, and low-traffic instances first, retaining an old-version control. Stop on parse errors, telemetry loss, export failures, CPU/RSS, backend write cost, or business p99 regression. If a language implementation is incomplete, keep the old path and register the gap rather than forcing cosmetic uniformity.
7. Roll back and govern long term
The launcher should support three states: new file, old variables, and explicitly disabled telemetry. A parse failure must not become an “empty but successful” SDK; fall back to the previous signed artifact or old variables and alert. Re-run the matrix on every SDK upgrade and retire environment variables gradually.
High-quality sample answer
I would start with a language and SDK capability matrix, separating a stable declarative schema from implementation maturity. The platform would provide versioned templates and small service-level overrides. After rendering, CI would check schema, permissions, endpoints, cardinality, and minimal startup, producing a signed immutable artifact. During migration, old environment variables remain an explicit fallback and the effective configuration is recorded so file and variable settings cannot create duplicate exporters. I would validate fixed telemetry samples for all four languages, then canary by language and business risk. Stop conditions include parse failures, drop rate, export failures, resource overhead, and cost regression. On file failure the launcher returns to the previous artifact or old variables, never to a silently uninstrumented instance; SDK upgrades rerun the matrix and advance deprecation.
Common mistakes
- Mistake → migrating every language because the schema is stable → stable specification does not mean complete fields in every implementation; fix: maintain a language and SDK capability matrix.
- Mistake → allowing files and variables without precedence → exporters or samplers can be instantiated twice; fix: declare precedence and emit a redacted effective snapshot.
- Mistake → checking only that the process starts → spans may be dropped, cardinality may explode, or data may reach the wrong endpoint; fix: compare counts and cost with fixed synthetic telemetry.
- Mistake → silently using an empty configuration after parse failure → an outage becomes an observability blind spot; fix: fall back to a signed previous artifact or old variables and alert.
Follow-up questions and responses
What if a language lacks a required exporter field?
Do not force uniformity. Keep that service on the old-variable path, record the field gap and risk, and migrate only through an audited adapter if necessary. Label the result partially compatible rather than pretending the full schema is supported.
How do you stop teams from overriding the platform sampling policy?
Allowlist overridable fields, run policy checks after rendering, and record the effective summary and change owner. The platform must enforce expensive exporters, sensitive endpoints, and sampling ceilings.
What if configuration loads successfully but backend cost doubles?
Slice by language, version, sampling, batching, retries, and resource attributes. Check duplicate exporters and high-cardinality labels first. Pause the canary, roll back the artifact, fix the configuration, and rerun the same synthetic workload instead of immediately adding backend capacity.