Prompt and applicable context
You own a GenAI application that retrieves documents, calls models, and executes tools. The team needs to know which model version raised time to first token, whether a tool increased failures, whether cost is concentrated in a few tenants, and whether a quality regression can be replayed without exposing personal data in prompts. Design traces, metrics, events, and data-quality gates using OpenTelemetry semantic conventions.
This is a telemetry-data modeling and validation problem, not a vendor-selection exercise. OpenTelemetry defines semantic conventions as shared attribute names and meanings across codebases, libraries, and platforms. GenAI conventions cover models, conversations, tool calls, token usage, and evaluations, but parts are still evolving, so not every attribute should be treated as a stable contract.
What the interviewer assesses
The interviewer wants signals derived from business questions, not a log full of model parameters. A strong answer correlates one user request with retrieval, model, tool, and final-result work while retaining enough version and tenant dimensions to explain a regression.
You must define data boundaries. Prompts, responses, tool arguments, and retrieved documents may contain sensitive data; high-cardinality session IDs, user IDs, or full content should not become unbounded metric labels. Collection must balance privacy, storage cost, sampling, and retention.
Finally, specify quality gates. If a parent span, model version, finish reason, or token count is missing, mark the record incomplete and monitor it instead of presenting a precise-looking dashboard.
Clarifying questions
First ask what outcome matters: time to first token, total latency, cost per request, tool success, retrieval recall, or answer quality. That choice determines which traces, metrics, and evaluation events are essential.
Then ask about sensitivity and jurisdiction. May prompts and responses be stored? Is regional isolation or tenant-level deletion required? If raw content is prohibited, use hashes, redacted summaries, reference IDs, and secure offline replay.
Finally ask about volume and retention. Request rate, model-call count, tool depth, sampling rate, and retention period determine collectors, queues, storage partitioning, and the cost envelope.
30-second answer framework
You can say:
“I would model each user request as one trace, with retrieval, generation, tool calls, and evaluation as versioned child spans or events. Metrics use low-cardinality dimensions such as model, provider, workflow, and result state. Prompts, responses, and tool arguments go to a controlled event stream with redaction, sampling, and audited access. Every record is checked for trace correlation, ordering, model version, and token counts; missing fields become data-quality failures. Dashboards connect latency, cost, errors, and quality scores, while a fixed sample of requests remains available for secure replay.”
Step-by-step deep answer
Draw the end-to-end signal graph
The entry span records request ID, a tenant summary, and workflow version. A retrieval span records source ID, query summary, and result count. A model span records provider, request model, response model, streaming flag, time to first token, and token usage. A tool span records tool type, call ID, and result state. An evaluation event records evaluator name, score, and an explanation reference.
Use low-cardinality metric dimensions
Aggregate latency, tokens, cost, and success by model, provider, workflow, region, result state, and error class. Keep session IDs, user IDs, document IDs, and full error text in traces or events rather than metric labels to avoid a time-series explosion.
Separate content events from runtime metrics
OpenTelemetry’s GenAI guidance describes traces, metrics, and events as three signals. Prompts and responses fit an event stream because they are large, sensitive, and subject to different retention. Runtime metrics keep counts, latency, and cost; content events use controlled storage, encryption, and shorter retention.
Record versions and causality
Each span records model, prompt-template, retriever, tool-definition, and evaluator versions. A regression investigation must connect a score to the exact model request and input-data version; one model name cannot distinguish alias drift, routing changes, and prompt changes.
Control sampling and cost
Keep complete traces at a fixed rate, then increase sampling for errors, extreme latency, low scores, and new versions. Aggregate token and tool-call counts in real time; enforce a tenant budget with throttling or degradation. Sampling rules belong in quality metadata so version-to-version comparisons remain valid.
Apply privacy and access controls
Redact fields at the SDK or collector so passwords, personal data, and keys never enter attributes. Authorize content events separately from runtime metrics, using tenant scope, encrypted references, and deletion indexes. “Internal only” is not a justification for retaining sensitive content.
Validate completeness and ordering
Check parent-child relationships, monotonic timestamps, finish state, model version, and tool-call IDs for every trace. For metrics, check units, bucket boundaries, monotonic counters, and duplicate reports. For events, check schema version, redaction state, and size limits. Send failures to a bad-data queue with a reason code.
Connect quality through evaluation events
An evaluation event contains evaluator name, score, labels, and an explanation reference. Do not treat an automated score as ground truth; record evaluator and dataset versions plus human samples. A quality regression should be reviewed with latency, cost, and errors rather than optimizing one signal alone.
High-quality sample answer
“I would represent each request as a correlated trace: entry, retrieval, model, tools, and evaluation are versioned spans or events. Metrics use low-cardinality model, provider, workflow, and result dimensions to aggregate time to first token, total latency, tokens, cost, and errors. Prompts, responses, and tool arguments go to a redacted, access-audited content stream rather than metric labels. The collection layer checks parent-child links, ordering, schemas, model versions, and token counts; invalid records enter a bad-data queue. Complete traces are fixed-sampled, while errors, slow requests, and quality regressions receive more sampling. Dashboards join runtime metrics to evaluation scores and replay data using dataset and evaluator versions.”
Common mistakes
Putting all content in metric labels
Failure pattern: using user IDs, session IDs, or full prompts as labels. Why it fails: high cardinality can overwhelm storage and queries while increasing privacy exposure. Correction: keep metrics low-cardinality and put content in controlled events.
Recording only the model name
Failure pattern: comparing every result by one model-name field. Why it fails: alias routing, prompt templates, retrievers, and tool versions become indistinguishable. Correction: record request model, response model, workflow, and dependency versions.
Optimizing token cost alone
Failure pattern: declaring success when cost falls. Why it fails: time to first token, tool success, or answer quality may degrade. Correction: monitor cost, latency, errors, retrieval, and evaluation together.
Storing raw prompts and responses
Failure pattern: retaining raw text indefinitely for replay. Why it fails: content can include personal data, keys, or cross-tenant information. Correction: redact fields, use short retention, encrypted references, tenant access, and deletion indexes.
Ignoring bad data
Failure pattern: including records with missing parents or token counts in the denominator. Why it fails: the dashboard appears complete but cannot explain its numbers. Correction: define completeness gates, a bad-data queue, and a quality dashboard that distinguishes missing from true zero.
Follow-up questions and responses
What if the team asks to record the full reasoning process?
Clarify whether the business needs auditable evidence, tool traces, or internal model reasoning. Prefer input references, tool calls, versions, evaluation evidence, and result summaries; do not retain unnecessary sensitive content or internal reasoning text.
What if the GenAI conventions migrate?
Store schema versions in collectors and warehouses, map old to new fields, and use dual writes or compatibility views during migration. Group dashboards by convention version instead of mixing meanings.
What if peak traffic makes complete traces too expensive?
Keep complete traces for errors and extreme latency, sample normal traffic at a fixed rate, and retain aggregate metrics plus event summaries. Record sampling rate, trigger rules, and drop reasons as quality metadata.
What if evaluation scores fall while latency improves?
Slice by model, prompt template, retrieval dataset, tool version, and tenant, and verify that the dataset and evaluator did not change. Let the release gate weigh quality regression against performance gains rather than accepting one metric.
How do you prove traces do not cross tenant boundaries?
Run tenant-boundary tests at collection, transport, storage, and query layers. Synthetic data should verify tenant scope on attributes, events, and references; anomalous queries emit audit alerts. Deletion requests must locate every derived event through an index.