Prompt and context
A multilingual site has public pages, authenticated pages, and temporary preview pages. The team added the preview path to robots.txt with Disallow, added noindex to the page, and used one canonical URL for every language. After launch, preview URLs still appear in search and some language pages have the wrong indexed URL. Explain the boundaries of the three signals and propose a migration and verification plan.
What the interviewer is testing
- Whether you separate crawling, indexing, and preferred-URL selection.
- Whether you know that a crawler blocked by robots.txt may not see noindex.
- Whether you can handle HTML meta robots, HTTP
X-Robots-Tag, canonical, redirects, and conflicts. - Whether you can design multilingual, cache, rollback, and Search Console checks.
Clarifying questions to ask first
- Is the goal to block crawling, block indexing, or consolidate duplicate URLs?
- Must a search crawler fetch the page to discover its noindex or canonical?
- Are preview pages authenticated, and can external links or sitemaps reference them?
- Are language pages one-to-one translations with their own canonical and hreflang?
- Where are these headers and tags set: CDN, application server, or HTML template?
A 30-second answer framework
I separate crawling, indexing, and canonicalization. robots.txt controls whether a crawler may request a resource; noindex controls whether a fetched page enters the index; canonical is a preference for duplicate or near-duplicate URLs, not a redirect. Preview pages should be authenticated or crawlable with noindex. Production language pages should self-canonicalize and cross-reference each other with hreflang. I would verify the rendered response, crawler access, cache behavior, and Search Console reports.
Step-by-step deep answer
Step 1: Define the three control planes
robots.txt at the site root answers whether a crawler may request a path. It does not delete an indexed URL and does not guarantee that a URL discovered through external links stays out of results. noindex is a page-level indexing directive for preview, search-result, or other non-public pages. canonical is a URL-selection signal for duplicate or near-duplicate content.
Step 2: Do not hide noindex behind Disallow
When a path is also Disallowed, a crawler may not fetch the page and therefore cannot see its meta robots tag or X-Robots-Tag: noindex. To make noindex observable, allow crawling and return noindex in HTML or HTTP headers. Sensitive content needs authentication or authorization, not crawler directives. During migration, remove the crawl block and wait for recrawling and index updates.
Step 3: Choose the page-level noindex carrier
HTML can use meta name="robots" content="noindex,follow"; non-HTML resources and gateways can use X-Robots-Tag. Test the template and response headers, and invalidate stale CDN objects. follow can preserve link discovery for a page intended to stay out of the index, but it is not access control and crawler behavior still applies.
Step 4: Design canonical and language mapping
Each indexable language page should generally have an absolute, reachable, self-referencing canonical with consistent status, scheme, host, and path. Do not canonicalize every translation to the default language or the translations may be consolidated as duplicates. Use hreflang for alternates, and make sitemaps, internal links, and canonicals use the same canonical URL set.
Step 5: Handle migration, redirects, and caches
Use server-side permanent redirects for URL migrations; an old page's canonical does not replace a redirect. Before submitting a sitemap, verify the new page's 200 response, canonical, robots directives, and hreflang. Invalidate CDN responses by URL and headers so edges agree. A rollback must preserve the old redirect and indexing policy rather than exposing mixed versions.
Step 6: Build an observable diagnosis path
Check robots.txt access first, then inspect the live HTML and response headers to compare noindex and canonical with the template. Next inspect Search Console crawl and index status, Google-selected canonical, last crawl time, and sitemap discovery. Log the URL, language, status, canonical, robots directives, cache age, and release version for auditability.
Step 7: Define acceptance and regression tests
Create assertions for each page type: previews are crawlable and noindex; production language pages return 200 and self-canonicalize; old URLs 301 to new URLs; protected pages do not leak content. Test robots rules for case and trailing-slash boundaries, HTML/header conflicts, multi-edge cache invalidation, and sampled Search Console states.
High-quality sample answer
The signals solve different problems: robots.txt controls requests, noindex controls index inclusion, and canonical suggests the preferred URL among duplicates. A preview page cannot be both blocked and expected to expose noindex; let crawlers fetch it with noindex, and use authentication for secrets. Each production language page should self-canonicalize and use hreflang, while sitemaps and internal links agree. Use 301 redirects for migrations, then verify robots, live responses, caches, crawl logs, and Search Console layer by layer.
Common mistakes
- Treating robots.txt as a guaranteed removal mechanism.
- Blocking a page and expecting crawlers to read its noindex.
- Canonicalizing every language page to one language.
- Checking source HTML while ignoring CDN headers and stale objects.
- Using canonical instead of a required redirect or access control.
Follow-up questions and responses
Follow-up 1: Why can a blocked URL still appear in results?
A crawler may discover the URL through external links but be unable to fetch its content. The search engine can still show the URL or limited information. To block indexing, make the page fetchable with noindex or require authentication.
Follow-up 2: Is canonical a mandatory directive?
It is a signal that can be ignored. Content, redirects, internal links, sitemaps, and canonical tags should agree; canonical is not access control or a deletion guarantee.
Follow-up 3: What if a preview must remain discoverable through links?
Allow crawling and return noindex. If the preview is sensitive, use login, signed URLs, or another access-control mechanism. Do not depend on robots.txt to hide secrets.
Follow-up 4: How do you resolve an HTTP-header and meta-robots conflict?
Generate both from one page-policy source and sample the origin and edge responses. Do not guess which signal won; inspect the actual response, HTML, and target search-engine documentation, then emit one explicit policy.
Follow-up 5: How do you test multilingual canonical loops?
Fetch every language URL and verify an absolute, reachable, self-referencing canonical. Then verify hreflang reciprocity and language codes, and check that sitemaps and internal links do not point to an old host or wrong language.