Prompt and context
A code-review tool, account system, or domain display receives visually similar characters, or bidirectional controls that change the reading order. Explain the difference between logical and display order, distinguish confusable detection from policy enforcement, and design a flow that preserves legitimate multilingual text.
What the interviewer evaluates
- Whether you understand that the Unicode bidirectional algorithm changes presentation order while logical code-point order remains intact.
- Whether you can distinguish overrides, isolates, mixed scripts, and confusable detection.
- Whether you know that the UTS #39 skeleton is an intermediate value that must not be displayed or reused across Unicode versions.
- Whether input validation, presentation, audit, authorization comparison, and upgrades are designed as separate concerns.
Clarifying questions to ask
First identify whether the field is source code, a login identifier, an internationalized domain, search text, or ordinary prose. Confirm allowed scripts, target languages, and display support. Then ask whether a finding blocks, requires review, warns, or is only logged. Confirm the Unicode data version, original-text retention, and acceptable false-positive rate.
30-second answer
Unicode text has a logical order and a display order. UAX #9 reorders presentation from directional properties, but bidi controls do not change comparison, parsing, or numeric analysis. Security policy should restrict dangerous controls and use UTS #39 for script and confusable checks. A skeleton is a versioned intermediate comparison key, not display text. Keep the original, attach versioned diagnostics, block or review high-risk identifiers, and use the field protocol’s exact comparison rules for authorization.
Step-by-step deep dive
1. Separate logical order from display order
When Arabic or Hebrew is mixed with digits, UAX #9 computes display order from strong, weak, and neutral directional types. Overrides such as RLO and LRO force direction, while isolates limit how an inner segment affects its surroundings. Rendering does not rewrite the code-point sequence in memory and must not become a parsing or comparison rule.
2. Identify bidi-control and mixed-script risk
Structured identifiers such as source code, filenames, and account names rarely need arbitrary directional controls. The input layer can reject or mark Bidi_Control characters; the display layer can show escapes or explicit boundaries. Mixed-script policy should follow the business’s allowed language set, not treat every non-ASCII character as malicious.
3. Apply versioned confusable detection
UTS #39 supplies confusables data and skeleton mechanisms for visual-confusability checks. Confusability depends on fonts, scripts, and context; it is not an absolute equivalence relation. Store the original, Unicode version, script analysis, and finding. Recompute after data upgrades and review new collisions, while authorization continues to use the protocol’s exact comparison rule.
High-quality sample answer
I would start by classifying the field’s risk. For structured identifiers such as source code, usernames, and domains, I would retain the original and restrict scripts and directional controls; ordinary multilingual prose should keep legitimate bidirectional layout. UAX #9 determines display order, while bidi controls must not alter parsing, comparison, or numeric analysis, so an audit tool should show both logical code-point order and rendered output. Detection would combine UTS #39 restriction levels, mixed-script rules, and confusables data. A skeleton is an intermediate key for one Unicode data version: it is neither display text nor a cross-version permanent identifier. RLO, an unusual script mix, or a collision with an existing high-risk identifier should block or require review; ordinary prose can produce a warning. Comparison and authorization must use the field protocol’s normalization, case, and encoding rules rather than visual similarity. On a Unicode data upgrade, recompute findings in a batch and review collisions so the change is traceable and reversible.
Common mistakes
- Treating display order as the stored string order.
- Claiming that deleting every right-to-left character solves the problem, breaking legitimate Arabic or Hebrew text.
- Treating a skeleton as a stable hash, display value, or cross-version protocol field.
- Checking only ASCII and ignoring fonts, scripts, combining marks, and context.
- Using visual similarity for authorization, signatures, or uniqueness instead of the field’s exact comparison rule.
Follow-up questions and responses
Why not delete bidi controls from every input?
They can support deception, but legitimate documents and layout may require directional controls. Choose blocking, escaping, isolation, or warning by field risk and target platform, and retain auditable evidence.
Can a skeleton become the username directly?
No. It is an intermediate detection form and changes with Unicode data. Keep the original and the protocol-defined comparison key; use the skeleton for collision diagnosis or review.
How do you reduce false positives in multilingual data?
Define allowed language and script sets, then combine CLDR language data, field context, and human review. Warn on ordinary prose, while applying stricter restrictions to login identifiers, domains, and code identifiers.