The Complete Guide to Crawling and Indexing Control (robots.txt / noindex / X-Robots-Tag / data-nosnippet / rel)

One-sentence mental model: Google has 3 independent operations:Crawl(whether to crawl) → Index(whether to index) → Serve(how to display). Each layer has different control tools,and mixing them up leads to self-contradiction.

TL;DR — 3-Layer Control Comparison Table ⭐⭐⭐

What you want	What to use	Where to implement
Prevent Google from crawling(save server resources)	`robots.txt` in the `Disallow:`	site root `/robots.txt`
Allow crawling but do not index in search results	`noindex`	`<meta name="robots">` or `X-Robots-Tag` HTTP header
Allow indexing but restrict presentation(limit snippet/preview/translation/expiration etc.)	`nosnippet` / `max-snippet:` / `max-image-preview:` / `notranslate` / `unavailable_after:`	meta tag or HTTP header
Allow indexing but exclude certain text within a section from snippets	`data-nosnippet` HTML attribute	Inside the page on `<span>` / `<div>` / `<section>`
Do not pass PageRank for a specific link	`rel="nofollow"` / `sponsored` / `ugc`	`<a>` tag
Completely not public(data security)	Password protection / delete	Server level

⚠️ Fatal classic pitfall

Using robots.txt Disallow on a page ≠ that page will not appear in search results.

Google will not crawl it, but if other sites link to it, Google may still write the URL into search results(with no description, only a bare URL). To truly prevent it from appearing → use noindex or password protection.

Worse: if you first Disallow in robots.txt and then add noindex → Google can never crawl this page → never sees noindex → this page keeps appearing in search results.robots.txt and noindex are mutually exclusive.

Part 1 — The real purpose of robots.txt

1.1 Main uses (2 things)

Manage crawl traffic — prevent certain paths from being crawled frequently to save server resources
(For media files only) Remove from search results — images/videos/audio must be blocked with robots.txt to not appear in Google Images/Videos

1.2 ⚠️ robots.txt Things it is not

❌ It is not a mechanism to hide pages
❌ It is not a security mechanism (Google follows the rules, but other crawlers may not)
❌ It is not a replacement for noindex

1.3 Behavioral differences by file type

File type	Effect of robots.txt Disallow
Web page(HTML/PDF/text)	⚠️ Only blocks crawling,does not block display: if other places link to it, the URL will still appear in search results (without a description). To completely prevent display use `noindex` or password protection.
Media files(images/video/audio)	✅ Truly will not appear in Google Images/Videos. But if other people's pages link to your image, they can embed and display it normally.
Resource files(JS/CSS/unimportant images)	⚠️ Only block when "the page is basically readable without these resources". If resources affect Google's understanding of the page → do not block, otherwise Google cannot render it → indexing quality drops.

1.4 3 fundamental limitations

Not all search engines comply — Google/Bing etc. follow the rules,many crawlers and AI scrapers do not
Syntax parsing differs across crawlers — the same rule may behave differently in Google and other engines
Disallowed pages can still be indexed — see the "fatal pitfall" above

1.5 Creating/updating

CMS users (WordPress/Wix/Blogger):Most CMS already handle this, do not modify manually. Find your CMS's "search settings" page.

Part 2 — `noindex`: The correct solution for not appearing in search results

2.1 Key prerequisite ⭐

noindex For it to take effect, the page must be crawlable.

❌ robots.txt Disallow this page → Google cannot see noindex → the page may still appear in results
❌ Added noindex but the page is down / returns 502 → Google also cannot see it → ineffective
✅ Must allow crawling + add noindex for it to work

2.2 Two implementation methods (choose one,do not use both)

A. <meta> tag(most commonly used,applicable to HTML pages)

<meta name="robots" content="noindex">

Targeting Google only:

<meta name="googlebot" content="noindex">

Note: Google does not require the meta tag to be placed in the <head>, placing it in the <body> is also recognized. But other search engines may not,so honestly place it in the <head> .

B. X-Robots-Tag HTTP response header(The only option for non-HTML files: PDF/video/images)

HTTP/1.1 200 OK
(...)
X-Robots-Tag: noindex
(...)

⚠️ Writing Noindex: in robots.txt is not supported by Google. Don't try it.

2.3 3 major reasons noindex is not taking effect

Google hasn't recrawled the page yet — low-importance pages may only be recrawled once every several months. Use URL Inspection tool to request a recrawl.
robots.txt blocked this page — see 2.1. Modify robots.txt to allow crawling.
noindex is not actually exposed to Googlebot — use the URL Inspection tool to see what HTML Googlebot actually received (not what you see in your browser).

2.4 Monitoring

Search Console → Page Indexing report can monitor which pages were removed due to noindex.

Part 3 — `robots` 11 valid rules for the meta tag ⭐⭐⭐

3.1 Complete rule table

Rule	Effect
`all`	Default, no restrictions
`noindex`	Do not index this page
`nofollow`	Do not follow links on this page (independent of indexing)
`none`	= `noindex, nofollow` shorthand
`nosnippet`	Do not show text snippets / video previews. Also prohibits content from being used as direct input for AI Overviews / AI Mode. Static image thumbnails may still appear (if helpful for UX)
`indexifembedded` ⭐	Allow indexing of versions embedded in other pages' iframes, even if this page has `noindex`. Must be used together with `noindex` to take effect
`max-snippet: N`	Snippets are at most N characters.`0` = nosnippet,`-1` = Google decides the length.Also limits how much AI Overviews/Mode can use
`max-image-preview: [none/standard/large]`	Image preview size
`max-video-preview: N`	Video snippets are at most N seconds.`0` = static image only,`-1` = no limit
`notranslate`	Do not provide translated versions in search results
`noimageindex`	Do not index images on this page (the same image appearing on other pages may still be indexed by those pages)
`unavailable_after: [date]`	Do not show in search results after expiration. Dates support RFC 822 / RFC 850 / ISO 8601

3.2 3 deprecated / no longer effective historical rules

Rule	Status
`noarchive`	No longer used — the cached link feature is gone
`nocache`	Never used
`nositelinkssearchbox`	No longer used — the sitelink search box feature is gone

3.3 Combining multiple rules

Comma-separated(equivalent):

<meta name="robots" content="noindex, nofollow">

Multiple meta tags:

<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">

Multiple different crawler names — Google takes the union of all negative rules:

<meta name="robots" content="nofollow">
<meta name="googlebot" content="noindex">

→ Googlebot treats this page as noindex, nofollow.

3.4 The strictest rule wins in conflicts

<meta name="robots" content="max-snippet:50, nosnippet">

→ nosnippet wins (stricter).

3.5 ⭐ structured data is not restricted by robots meta

max-snippet and other restrictions only apply to content Google automatically extracts from the page.Content actively declared via structured data bypasses these restrictions.

To limit the description length of a recipe rich result → modify the structured data itself,not the robots meta
To limit the length of article.description / other creative work description → use max-snippet(this is an exception, because Google treats these two fields as snippets)
data-nosnippet structured data blocks wrapped in can still be used in search results(not excluded)

Part 4 — `data-nosnippet`: Element-level fine-grained control

4.1 Effect

Prevent a specific segment of text within a page from entering snippets A certain segment of text will not appear in snippets, but the page as a whole can still be indexed.

4.2 Syntax rules

Can only be used on <span> / <div> / <section>
It is a boolean attribute — writing data-nosnippet, data-nosnippet="", data-nosnippet="true", data-nosnippet="false" are all treated as enabled(characteristic of boolean attributes)
HTML must be valid — tags must be properly closed, otherwise when Google parses,it may exclude all subsequent content as well

4.3 Examples

<p>This text can be shown in a snippet
<span data-nosnippet>and this part would not be shown</span>.</p>

<div data-nosnippet>not in snippet</div>
<div data-nosnippet="true">also not in snippet</div>
<div data-nosnippet="false">also not in snippet</div>
<!-- 所有值都被视为 "打开" -->

4.4 ⚠️ Pitfalls of dynamically adding data-nosnippet with JS

Google data-nosnippet may be extracted both before and after rendering. Do not use JS to dynamically add/remove the data-nosnippet attribute(because during the pre-rendering extraction, your JS hasn't run yet).

Correct approach: Include data-nosnippet when the DOM element is first written to the page, and do not modify it afterwards.

Custom elements: Must be wrapped or rendered with <div>/<span>/<section> wrapped or rendered before you can use data-nosnippet.<mytag data-nosnippet> Invalid.

Part 5 — `X-Robots-Tag` HTTP response header

5.1 Why use it

The only option for non-HTML resources(PDF/images/video), because you cannot write <meta> tags
When you want to apply rules uniformly across the whole site rules across the whole site Easier to maintain than adding meta tags page by page
Supports regex matching of files → flexible

5.2 Configuration examples

Apache — noindex+nofollow for all PDFs across the site:

<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

NGINX — same as above:

location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}

Apache — noindex for all images across the site:

<Files ~ "\.(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>

Single file:

# .htaccess 必须放在文件所在目录
<Files "unicorn.pdf">
Header set X-Robots-Tag "noindex, nofollow"
</Files>

5.3 Combining multiple X-Robots-Tag headers

Two equivalent ways to write it:

X-Robots-Tag: noimageindex
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST

Or:

X-Robots-Tag: noimageindex, unavailable_after: 25 Jun 2010 15:00:00 PST

5.4 Specifying user-agent

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow

Not specifying a user-agent → applies to all crawlers.

HTTP header names, user-agent names, and rule values are all case-insensitive.

Part 6 — Pitfalls of using robots.txt + meta robots together ⭐⭐⭐

6.1 Core conflict

If a URL is Disallowed by robots.txt:

Googlebot It can never be crawled → Google never sees <meta name="robots" content="noindex"> or X-Robots-Tag: noindex
→ This page may still be indexed(based on external links, without a description)
→ You think adding noindex solves everything,but in reality it is completely ineffective

6.2 Decision tree

要不要 noindex?
├─ 是 → robots.txt 必须 Allow 这个 URL (或不写 Disallow)
│         → 加 <meta robots noindex> 或 X-Robots-Tag: noindex
│         → 等 Googlebot 重爬看见
│
└─ 只是省服务器流量(不在乎索引)
          → robots.txt Disallow 就好
          → 接受这页 URL 可能仍出现在结果里(无 description)
          → 接受其上的图/视频/PDF 也不会被爬

Part 7 — Decision table for 4 methods of controlling content

From control-what-you-share.Sorted from strongest to weakest.

Method	Applicable to	Advantages	Disadvantages
1. Delete content	All types	100% will not appear on Google or anywhere on the internet	Irrecoverable
2. Password protection	All types	True access control, Google cannot find it either	Users need to log in
3. `noindex`	All types	Content can still be accessed by users, but does not enter search results	Only takes effect after Googlebot crawls it; does not prevent other search engines (others may not support noindex)
4. robots.txt Disallow	Only effective for images/video/audio(for web pages it only blocks crawling, not indexing)	Simple	For web pages useless, only controls crawling
5. Opt out of specific Google products	Web pages only	Fine-grained (only blocks Shopping/Hotels/vacation rentals etc.)	Only works for Google, does not prevent other engines

7.1 When should you block content?

Data confidentiality — Data only accessible after user login (note that file metadata may also leak, see keep-redacted-information-out)
Low-quality content(user-generated spam) — indexing these will drag down the entire site's rankings
Crawl budget optimization for large sites — For sites with > hundreds of thousands of URLs, prevent Google from wasting crawl on unimportant / duplicate content

7.2 How to remove already indexed content

Use Remove information. Search Console Removals tool to temporarily (~6 months) remove it from search results, buying time to handle it thoroughly.

Part 8 — Special section: Removing images

8.1 Emergency removal

Search Console Removals tool → Select the image URL. ⚠️ Temporary(default ~6 months), it will come back after expiration. For permanent removal → configure robots.txt or X-Robots-Tag noindex as a fallback.

8.2 Two methods for non-emergency removal (choose one)

A. robots.txt Disallow + Googlebot-Image UA

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

Remove only dogs.jpg without affecting other images.

Multiple images:

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
Disallow: /images/cats.jpg
Disallow: /images/llamas.jpg

# 或用 wildcard
Disallow: /images/animal-picture-*.jpg

Remove all images across the entire site:

User-agent: Googlebot-Image
Disallow: /

Remove all images of a certain format(example: remove all.gif files):

User-agent: Googlebot-Image
Disallow: /*.gif$

8.3 Which UA to choose?

UA	Affects
`Googlebot-Image`	Only removed from Google Images (will still appear in web search)
`Googlebot`	All Google Search removed entirely

8.4 B. `X-Robots-Tag: noindex` HTTP header

Suitable for cases where images are hosted on a CDN or robots.txt cannot be modified.Prerequisite: The image must be crawlable by Googlebot (so that the header can be read).

8.5 `noimageindex` limitations

Page-level <meta name="robots" content="noimageindex"> prevents images embedded on that page from being indexed,but if the same image appears on other pages, it will still be indexed via those other pages.

Truly globally block an image from being indexed → Use X-Robots-Tag: noindex.

Part 9 — User-Triggered Fetchers (9 total,ignore robots.txt)

Key point: These fetchers are triggered by users'active actions(not crawled by Google on its own), so they usually ignore robots.txt. Your disallow rules will have no effect.

9.1 Complete list

Name	UA token	Purpose
Chrome Web Store	`Google-CWS`	Crawls URLs in Chrome extension metadata
Feedfetcher	`FeedFetcher-Google`	Crawls RSS/Atom feeds for Google News + WebSub
Google-Agent ⭐	`Google-Agent`	Used by Google AI agents (e.g., Project Mariner). Uses the user-triggered-agents.json IP range. Google is also testing Web Bot Auth using the `https://agent.bot.goog` identity.
Google Messages	`GoogleMessages`	Generates previews for URLs in chats
Google NotebookLM	`Google-NotebookLM`	Source URLs added by users in NotebookLM
Google Pinpoint	`Google-Pinpoint`	Document sources added by users in Pinpoint
Google Publisher Center	`GoogleProducer`	Feeds actively provided by publishers to Google News
Google Read Aloud	`Google-Read-Aloud`(formerly `google-speakr` is deprecated)	User-requested TTS reading aloud
Google Site Verifier	`Google-Site-Verification/1.0`	Search Console ownership verification

9.2 Reverse DNS verification

Google-owned fetchers: ***-***-***-***.gae.googleusercontent.com
User-hosted (GAE user sites): google-proxy-***-***-***-***.google.com

9.3 ⚠️ UA can be spoofed

To confirm a request is really from Google → perform verify Google requests reverse IP + forward DNS verification.

Part 10 — Outbound link qualifiers (`rel` attribute)

10.1 4 types of rel values

`rel` Value	Purpose	Example
(not written)	Normal link, Google follows and parses it normally	`<a href="...">x</a>`
`sponsored`	Advertisements / paid promotions(paid link)	`<a rel="sponsored" href="...">`
`ugc`	User-generated content(comments / forums)	`<a rel="ugc" href="...">`
`nofollow`	Other links you do not want to be associated with or do not want followed	`<a rel="nofollow" href="...">`

10.2 About `sponsored` vs `nofollow`(evolved in 2019)

Before 2019: paid links used nofollow
After 2019: recommends sponsored, but nofollow is still acceptable
Do not miss marking — unmarked paid links will trigger link spam policy

10.3 About `ugc`

Recommended for comments / forums / message boards
You may selectively remove rel="ugc" from links belonging to "long-term high-quality contributors" as a reward

10.4 Combining multiple rel values

Spaces or commas both work:

<a rel="ugc nofollow" href="...">x</a>
<a rel="ugc,nofollow" href="...">x</a>

10.5 ⚠️ Key points

These rel attributes are hint, Google usually does not follow these links,but:
- the linked page may be discovered through other channels (sitemap / other external links)
- so rel="nofollow" does not prevent a page from being indexed
To truly prevent Google from crawling a page on your own site → Use robots.txt 's Disallow:,Do not rely on nofollow
To truly prevent a page on your own site from being indexed → Use noindex,Do not rely on nofollow
rel="nofollow" can also be used as a robots meta tag (<meta name="robots" content="nofollow">), other rel values cannot

Part 11 — Practical decision tree

11.1 I want to protect data for privacy / security

→ Password protection(server-side auth). robots.txt / noindex / nofollow are all unreliable.

11.2 I have a test environment / staging server

→ Password-protect the entire site(basic auth works). Do not rely on robots.txt — if you forget to change it after launch, Google will index your staging URLs and you will never fully clean it up.

11.3 I don't want this PDF to be found in search

→ Configure the HTTP server X-Robots-Tag: noindex for.pdf files.robots.txt is useless(PDFs are text-type; even if disallowed, they may still be indexed via external links).

11.4 I don't want this image to be found in search

→ robots.txt User-agent: Googlebot-Image Disallow: /... or on that image URL X-Robots-Tag: noindex. Choose one.

11.5 I have 50 links in comments and I don't want Google to pass PageRank

→ Add rel="ugc"(or rel="nofollow") to comment links. Most CMSes add this automatically.

11.6 I have 5 ad slots I want to mark compliantly

→ Add rel="sponsored".

to ad links

→ <meta name="robots" content="unavailable_after: 2026-12-31T00:00:00+08:00">

11.7 I have an article that is about to go behind a paywall and I want to remove it from search results

→ Do not use max-snippet restrictions; instead directly edit the article.description schema field.

11.8 I don't want the description on this page to be truncated too short (rich results use the schema description)

#	Anti-pattern	Consequence
1	robots.txt Disallow a page + expect it not to be indexed	It will still appear in results (with no description)
2	robots.txt Disallow + add noindex	noindex will never be seen,they conflict with each other
3	Writing in robots.txt `Noindex: /xxx`	Google does not support this directive, it is completely ineffective
4	robots.txt blocks important JS/CSS	Rendering fails, indexing quality decreases
5	Using `noarchive` / `nocache` / `nositelinkssearchbox`	is deprecated, Google completely ignores it
6	Dynamically adding/removing via JS `data-nosnippet`	The extraction performed before rendering will see an inconsistent state, behavior is unpredictable
7	`data-nosnippet` using on `<custom-tag>`	is invalid, only span/div/section are supported
8	Expecting robots.txt to prevent malicious crawlers	It is not a security mechanism, only compliant crawlers obey it
9	Expecting nofollow to completely prevent a page from being indexed	nofollow is only a hint, the page can still be discovered
10	Paid links not marked with `rel="sponsored"` or `nofollow`	Violates link spam policy, may be penalized
11	Trying to prevent images from being indexed via `robots.txt` Block `noimageindex` to prevent images from being indexed	noimageindex is page-level, placed on the page rather than the image URL, and only prevents images embedded on that page
12	Using `<meta name="ai-bot" content="noindex">` to try to block AI crawling	Google does not recognize `ai-bot` this custom UA token; it only recognizes `robots` / `googlebot` / `googlebot-news`
13	meta robots not placed in `<head>` expecting failure as a "way to test"	Google There is no mandatory head/body position, it will still be read,you cannot use wrong placement as an undo button

Part 13 — Related wiki

Syntax details of robots.txt (full spec): see Google robots.txt spec(not covered in depth in this wiki)
How to debug noindex with Search Console: Search Console + Analytics + Trends three-piece suite usage (traffic diagnostics / keyword research / Looker Studio) 's URL Inspection / Page Indexing report
Let Google see the pages you want indexed (reverse): Google SEO Basics + How Search Works + Search Essentials + Complete Guide to Sitemaps (XML / RSS / Text + image / video / news / hreflang extensions + sitemap index)
Crawling details / Googlebot behavior / Crawl stats: Complete Guide to Googlebot Crawling Mechanism and Indexing Control (URL/Links/Mobile-first/Meta/Testing/AMP)(future wiki)
Removing indexed pages + temporary takedown: Complete Guide to Security / Spam / Removal / Search Operators (malware + phishing + UGC + redacted + site:)(future wiki)
Links shown to crawlers must be crawlable: Full JavaScript SEO (basics + diagnostics + Dynamic Rendering + Lazy Loading) 's SPA / fragment URL / Local Storage related pitfalls

Last updated

2026-05-17 — Compiled based on 7 Google Search Central robots/control series articles (latest from 2025-12 ~ 2026-05). Includes the 2026-03 robots-meta-tag update (covering the relationship between AI Overviews / AI Mode and nosnippet) and the 2026-05 addition of Google-Agent (Project Mariner) + Web Bot Auth to user-triggered fetchers.