The Complete Guide to Googlebot Crawling Mechanisms and Indexing Control (URL/Links/Mobile-first/Meta/Testing/AMP)

In one sentence: How Googlebot crawls / how to write URLs / how to write links / how to configure mobile-first / which meta tags are useful / what a valid HTML head is / how to diagnose crawl issues / how to run A/B tests without breaking SEO / how to use and remove AMP — a collection of "foundational internal skill" knowledge for site crawling and indexing layers.

TL;DR — 9 core do/don't points

Do	Don't
Use `<a href="...">` real links	`<span href>` / `routerLink` / pure onclick
Use hyphens to separate words in URLs(`green-dress`)	underscore(`green_dress`)
Keep URL casing consistent	`/APPLE` and `/apple` are two different URLs
Mobile site and desktop must have equivalent content	Thin content on mobile version (Google only looks at mobile)
`<head>` Only use the 8 whitelisted tags in	Stuff `<iframe>` / `<img>` into head (cuts off subsequent reading)
Use `description` / `viewport` and other 9 supported meta tags	Use `meta-keyword`(completely ineffective)
For A/B testing use 302 + rel=canonical	Use 301 or noindex
For emergency overload use `503 / 429`(< 2 days)	Long-term 503 → URL is permanently removed from the index
Returning a 200 status code for an error page = soft 404	Google marks it as soft 404 and does not index it

Part 1 — Who Googlebot is, and how it crawls

1.1 Two subtypes

Type	UA token	When used
Googlebot Smartphone	Same user-agent token	Primary crawler — majority of requests
Googlebot Desktop	Same user-agent token	Minority — for desktop-only content compatibility

⚠️ In robots.txt both use the same token(both are Googlebot)→ You cannot block only one of them.

⚠️ Google mobile-first indexing → For most sites the primary indexing source is the mobile crawler.

1.2 Crawl limits

Resource	Limit
HTML / text	First 2MB(before decompression)
PDF	First 64MB
Resources (CSS / JS / etc.)	2MB per independent resource(fetched separately during rendering)
Googlebot Image / Video	Independent limit

⚠️ Exceeding the cutoff → truncated, only the downloaded portion is used.

1.3 Crawl frequency

For general sites once every few seconds on average
May be slightly higher in the short term (queue delay)
If the server cannot handle it → reduce crawl rate

1.4 Time zone

When crawling from US IPs, Googlebot uses Pacific Time(PT). Pay attention to time zone conversion when comparing logs.

1.5 Verify Googlebot (prevent UA spoofing)

UA is easy to impersonate,verification method:

Reverse DNS lookup on source IP (via verify Googlebot)
or match Googlebot IP ranges

1.6 3 options when blocking Googlebot

Purpose	What to use
Do not crawl this page	`robots.txt`
Do not index this page	`noindex`
Do not allow anyone / crawler access	Password protection

Part 2 — File types Google can index

2.1 Flat text types (unencoded text)

CSV / KML (Google Earth)/ KMZ / GPX (GPS)/ HTML / SVG / TeX/LaTeX / Text (.txt etc.)/ WAP / XML

Source code also count as text: BAS / C/C++(.c.cc.cpp.cxx.h.hpp)/ C#(.cs)/ Java(.java)/ Perl(.pl)/ Python(.py)

2.2 Encoded document types

Adobe PDF / PS
EPUB e-books
Korean Hancom Hanword(.hwp)
Microsoft Excel / PowerPoint / Word
OpenOffice odp / ods / odt
Rich Text RTF

2.3 Media

Image: BMP / GIF / JPEG / PNG / WebP / SVG / AVIF(new)
Video: 3GP / 3G2 / ASF / AVI / DivX / M2V / M3U / M3U8 / M4V / MKV / MOV / MP4 / MPEG / OGV / QVT / RAM / RM / VOB / WebM / WMV / XAP

2.4 Detection method

Content-Type HTTP header takes priority; if missing / incorrect, fall back to file extension / re-parsing.

filetype:rtf galway operator can restrict searches to RTF files containing "galway".

Part 3 — URL Structure Best Practices ⭐⭐⭐

3.1 Hard rules (must follow)

Rule	Explanation
Comply with IETF STD 66	reserved characters must be percent-encoded
Do not use fragments to change content	`#/potatoes` this kind — Google does not support it. For JS content changes use History API
URL parameters use `=` + `&`	Do not use `:` / `[]` / double commas
Separate multiple values with `,` Separator	`?color=purple,pink,salmon`

3.2 Soft rules (recommended)

Rule	Example
Descriptive URLs(readable words)	✅ `/wiki/Aviation` ❌ `/index.php?topic=42&area=3a5ebc944f41`
Use the target language	German site `/lebensmittel/pfefferminz`, Japanese site `/ペパーミント`
Percent-encode non-ASCII	In href use `/gem%C3%BCse` instead of `/gemüse`
Hyphen word separation(not underscore)	✅ `/summer-clothing` ❌ `/summer_clothing`
Fewer parameters	Remove parameters that do not change content
Case sensitivity	`/APPLE` ≠ `/apple` — when the server is case-insensitive, standardize on one casing
Use country codes for multi-region	`example.de` or `example.com/de/`

3.3 4 common URL problems

#	Problem	Example	Fix
1	Faceted filter combinatorial explosion	Hotel + beach + gym = 3-dimensional filter full combinations generate massive URLs	Faceted navigation management
2	Irrelevant parameters	referral id / sort parameters / session id	Replace session id with cookies; block irrelevant parameter URLs in robots.txt
3	Infinite calendars	Calendar clickable all the way to year 2999	Add `nofollow`
4	to future date links	`../../category/stuff` generates infinite loop URLs on error page paths	Use root-relative paths, do not use parent-relative paths

Part 4 — Link Best Practices

4.1 The only reliably crawlable link form

✅ Recommended(Google can parse):

<a href="https://example.com">x</a>
<a href="/products/category/shoes">x</a>
<a href="./products/category/shoes">x</a>
<a href="/products/category/shoes" onclick="...">x</a>  <!-- href 是主,onclick 不重要 -->
<a href="/products/category/shoes" class="...">x</a>

❌ Not recommended(may not be parsed):

<a routerLink="products/category">x</a>      <!-- Angular framework 私有属性 -->
<span href="https://example.com">x</span>     <!-- 不是 <a> 元素 -->
<a onclick="goto('https://example.com')">x</a>  <!-- 没有 href -->
<a href="javascript:goTo('products')">x</a>   <!-- href 是 JS 不是真 URL -->

⚠️ Dynamically inserted JS links also work — as long as the final HTML is in <a href> form.

4.2 5 anchor text guidelines

A. Do not use empty anchors

❌ <a href="https://example.com/page"></a>

When there is a title attribute, Google will use the title as a fallback; when an image is the anchor, use alt.

B. Be descriptive (do not use "Click here")

❌ Click here to learn more
❌ Read more
❌ Learn more about our cheese on our website
❌ We have an article about...

✅ For a full list of cheese available, see the list of cheese types

Self-check: Read the anchor text on its own,do you know what you get when you click? If not = too vague.

C. Be concise

❌ <a>Knitted Cow invites local residents of Wisconsin to their grand re-opening by also offering complimentary cow-shaped ice sculptures</a>
✅ <a>Knitted Cow invites local residents of Wisconsin</a> to their grand re-opening...

D. Do not keyword stuff

Violates spam policy.

E. Do not stack links

❌ <a>so</a> <a>many</a> <a>times</a> <a>this</a> <a>year</a>

Readers cannot distinguish them, context is lost.

4.3 Internal / external link strategy

Internal links: every important page has at least one other page pointing to it
External links: linking out is not scary, citing sources builds credibility.nofollow Only use when you do not trust the other party; for paid/UGC use sponsored/ugc
There is no "ideal number of links",If you feel it is too many, it is too many

Part 5 — Mobile-First Indexing ⭐⭐⭐

Google Primarily use the mobile version for indexing and ranking. This is mobile-first.

5.1 3 mobile configurations

Configuration	Description	Google recommendation
Responsive design	Same HTML same URL, CSS determines layout	⭐ First choice, easiest to maintain
Dynamic serving	Same URL, UA sniffing returns different HTML	Requires `Vary: user-agent` HTTP header
Separate URLs(m-dot)	Different URLs(`m.example.com`)	Most complex, requires canonical / alternate pairing

5.2 5 non-negotiable iron rules

Mobile and desktop content must be equivalent — not just visually,text content must also match. Thinned mobile content = lost indexing signals.
Mobile and desktop use the same robots meta tag — adding noindex on mobile = entire site completely not indexed after mobile-first.
Do not lazy-load main content until after user interaction — Google No scrolling / no clicking / no typing.
Structured data required on both versions — priority: Breadcrumb / Product / VideoObject. URLs must also be correct.
title and meta description consistent across both versions.

5.3 Separate URL configuration details (m-dot)

canonical configuration

Desktop version https://example.com/:

<link rel="canonical" href="https://example.com/">
<link rel="alternate" media="only screen and (max-width: 640px)" href="https://m.example.com/">

Mobile version https://m.example.com/:

<link rel="canonical" href="https://example.com/">

Core: desktop is always canonical, mobile is alternate.

hreflang configuration (each points to the other)

Desktop version:

<link rel="canonical" href="https://example.com/">
<link rel="alternate" media="only screen and (max-width: 640px)" href="https://m.example.com/">
<link rel="alternate" hreflang="es" href="https://example.com/es/">  <!-- desktop hreflang 指 desktop -->
<link rel="alternate" hreflang="fr" href="https://example.com/fr/">

Mobile version:

<link rel="canonical" href="https://example.com/">
<link rel="alternate" hreflang="es" href="https://m.example.com/es/">  <!-- mobile hreflang 指 mobile -->
<link rel="alternate" hreflang="fr" href="https://m.example.com/fr/">

Other separate URL rules

mobile version do not use URL fragments(Google does not index fragments)
desktop and mobile each have error page status codes consistent
Different desktop pages do not all redirect to the mobile homepage(all pages will disappear from the index)
Verify both versions in Search Console
mobile server capacity capacity sufficient

5.4 13 major Troubleshooting Errors

If Search Console reports these → mobile-first indexing interrupted / ranking drops:

Error	Fix
Missing structured data(missing on mobile)	Sync both versions, verify URLs, check Data Highlighter regularly
noindex tag	Mobile uses noindex = entire site indexing interrupted — remove it
Missing image(missing images on mobile)	Equivalent content + do not lazy-load main images + supported image formats
Blocked image(robots.txt block)	Allow Googlebot to crawl images
Low quality image(too small / low resolution)	Use high-quality images
Missing alt text	Add the same alt on mobile as on desktop
Missing page title	Add `<title>` on mobile equivalent to desktop
Missing meta description	Same as above
Mobile URL is an error page	Status codes consistent across both versions
Mobile URL has anchor fragment	Remove fragments, use History API
Mobile page blocked by robots.txt	Check robots.txt consistency
Duplicate mobile page target(multiple desktop → same mobile)	Each desktop page maps to an independent mobile page
Desktop redirects to mobile homepage	Each page maps to an equivalent mobile page
Page quality issues(too many ads / missing content)	Comply with Better Ads Standard
Video issues	Easy-to-find placement + do not lazy-load main video
Hostload issues(insufficient mobile server capacity)	Scale up capacity

Part 6 — Complete table of Meta Tags supported by Google

6.1 Basic structure

<meta> placed in <head>.HTML standard also supports XHTML form.

6.2 ✅ 9 meta tags supported by Google

Tag	Purpose
`description`	Page summary (may be used as a snippet)
`robots` / `googlebot`	Indexing + presentation control (see Complete Guide to Crawling and Indexing Control (robots.txt / noindex / X-Robots-Tag / data-nosnippet / rel) 11 valid rules)
`notranslate`	Do not provide translated versions in search results
`nopagereadaloud`	Disable Google TTS read-aloud
`google-site-verification`	Search Console ownership verification — name + content case must match exactly
`Content-Type` / `charset`	Content type + character set (UTF-8 recommended)
`refresh`	meta refresh — Not recommended, use server-side 301
`viewport`	Mobile rendering settings (presence = tells Google "mobile-friendly")
`rating`	adult content labeling(`rating="adult"` or RTA string)

6.3 ❌ 4 tags Google does not support (completely ineffective)

Tag	Status
`<meta name="keywords">`	Completely ineffective — Google stopped using it long ago
`<html lang="...">` attribute	Google looks at main body content to determine language, does not look at the lang attribute
`<link rel="next">` / `rel="prev">`	No longer used, completely ignored
`nositelinkssearchbox`	Feature retired, tag invalid

6.4 ⚠️ Rules for modifying meta with JS

Strongly recommend avoiding JS injection / modification of meta tags. If you must use it → use URL Inspection Tool to test the rendered HTML.

Part 7 — Valid HTML `<head>` element whitelist ⭐

⚠️ Core pitfall: <head> contains non-whitelisted elements → Google cuts off parsing starting from that element, subsequent meta / link / title are all ignored.

7.1 ✅ Whitelist (8 items)

title / meta / link / script / style / base / noscript / template

7.2 ❌ Common incorrectly inserted elements (cut off head)

<iframe>
<img>

Must use? Place it after all valid elements you want Google to read.

Part 8 — Troubleshoot Crawling Errors

8.1 5-step method

1. 看 Googlebot 是否遇到 availability 问题(host 抗不住)
2. 看是否有该爬却没爬的页
3. 看更新是否被及时爬到
4. 改进 crawl efficiency
5. 处理 overcrawling 紧急情况

8.2 Key tools

Crawl Stats report(Search Console) — host availability chart, check if Googlebot requests exceed the red line
URL Inspection tool — single URL testing,Hostload exceeded warning = server cannot handle the load
Page Indexing report — find soft 404 / index status

8.3 What to do when the server cannot handle the load

Block unimportant resources with robots.txt (crawl budget)
Speed up page loading
Scale capacity — if Crawl Stats stay above the red line but important URLs remain uncrawled, wait 1 month to see if improvement occurs

8.4 Important pages not crawled

Submit via sitemap
Check that robots.txt does not mistakenly block them
Optimize crawl budget, see manage your inventory
Check server capacity

⚠️ New pages typically take several days to be crawled — do not expect same-day indexing,unless you run a news site or highly time-sensitive site.

8.5 Help Google discover updates faster

Do	Avoid
News sites use news sitemap	Submit unchanged sitemaps daily
Use `<lastmod>` in sitemaps to indicate update time	Expect Googlebot to immediately crawl everything in the sitemap
Use crawlable URL structure + `<a href>` link	Put URLs you do not want indexed into the sitemap

8.6 If-Modified-Since / 304 optimization ⭐

Google occasionally sends If-Modified-Since header (especially AdsBot):

Server Return 304 Not Modified with no body → Google reuses the previously fetched content,saving server resources
If content is unchanged you may proactively return 304(no need for Google to ask first), which also saves resources

8.7 Fixing soft 404s

Soft 404 = page returns 200 but content is an error / empty.After Google's algorithm identifies this, it is removed from the index.

Status	Fix method
Page is truly gone	Return 404 or 410(404 + friendly prompt + internal links + report button)
Page has moved	Return 301 redirect to the new location
Page still exists but was misjudged	Check rendering with URL Inspection — most often caused by failed resource loading / blocked resources

8.8 Emergency handling of overcrawling

Server overwhelmed by Googlebot:

Temporarily return 503 or 429(retry-after) — hold for 2 days
Recover within 2 days — otherwise URLs will be permanently removed from the index
Monitor crawl rate + host capacity
If it is AdsBot crawling (once every 3 weeks), reduce Dynamic Search Ad targets or scale capacity

8.9 Hide URLs you do not want shown in Search (crawl budget optimization)

robots.txt block (faceted nav / calendars / internal search / shopping cart / "buy now")
Consolidate duplicate content with canonical
Replace session IDs with cookies

⚠️ Do not frequently add/remove robots.txt entries as a budget adjustment method — Google will get confused. robots.txt is for long-term content you do not want crawled.

Part 9 — Website Testing (A/B / Multivariate)

9.1 4 iron rules

Do	Reason
Do not cloak	Serving one set of content to Googlebot and another to users = violates spam policy, may result in removal from index
Use rel=canonical instead of noindex to mark test variants	noindex blocks completely, which is excessive; canonical indicates "this is a variant, the primary version is X"
Use 302 instead of 301 redirect	302 = temporary, Google keeps the original URL in the index; 301 = permanent and will replace it
After testing ends clean up immediately	Running long-term → Google treats it as deception, applies penalty

9.2 Two testing approaches

Multiple URLs: one URL per variant, split traffic via 302 / JS.All variants use rel=canonical pointing to the original URL.
Single URL dynamic JS: same URL uses JS to decide which variant to display. Googlebot does not send cookies → sees the no-cookie version.

9.3 Small tests you do not need to worry about

Button color / font / "Add to cart" vs "Buy now!" → even if Google sees them, they do not affect ranking / snippet.

Part 10 — AMP overview

⭐ AMP is no longer a core SEO priority(Top stories no longer requires AMP, after 2024). But if your site still has AMP pages you still need to manage them.

10.1 How AMP appears in Search

Form	Description
Rich result	Like regular pages, may appear in carousel
Web Stories	Special handling for Story format
Google AMP Viewer(mobile)	Shows publisher domain label at top; visits google.com/amp/...
Signed Exchange(supported browsers)	Uses publisher's own URL, supports first-party cookies + analytics — takes priority over Viewer

⚠️ AMP itself is not a ranking factor. Speed is.

10.2 Two configurations

Configuration	Meaning
Canonical AMP	Only one version, AMP itself is the canonical
Canonical non-AMP	Dual version, with non-AMP primary version + AMP sub-version

10.3 5 major guidelines

Must comply with AMP HTML spec
AMP content ≈ canonical content (users can complete the same actions)
URL path related: amp.example.com/giraffes or example.com/amp/giraffes(cannot be test.com/giraffes)
AMP must be valid (use AMP Test Tool)
structured data complies with SD policy

10.4 3 paths to remove AMP

Path	Steps
A. Remove entire page(AMP + non-AMP)	1) Delete both versions 2) Remove outdated content tool 3) Update AMP Cache 4) Verify
B. Remove AMP only(keep non-AMP)	1) Remove the `rel="amphtml"` link on the canonical 2) AMP URL returns 301 or 302 to canonical 3) Optional: for complete removal → 4) Return 404 + 5) update cache + 6) 301 permanent redirect
C. CMS operations	WordPress / Drupal / SquareSpace each have plugin help

⚠️ When deleting AMP do not directly empty the content — empty / markup-free document = invalid → Google will continue to serve the old version!

10.5 AMP deprecated vs still useful

Still useful	Deprecated
Speed advantage (AMP templates are inherently fast)	Top stories carousel no longer requires AMP(AMP requirement removed in 2024)
Signed Exchange + AMP Cache prerendering	AMP-only ranking boost (never existed)

Part 11 — Key anti-pattern collection

#	Anti-pattern	Consequence
1	`<span href>` / `<a onclick>` as links	Google does not parse them, may not be able to crawl at all
2	URLs use fragments(`#/page`)	Not indexed
3	URLs use underscores for word separation	Google does not recognize them as word separators
4	Inconsistent URL capitalization	Treated as different pages, causing duplicate content
5	Mobile version has stripped-down content	After mobile-first indexing rankings plummet
6	Mobile version has `noindex`	Entire site disappears from index
7	`<head>` contains `<iframe>` / `<img>`	Google cuts off head parsing, subsequent meta tags all fail
8	Using `<meta name="keywords">` expecting SEO improvement	Completely ineffective
9	Using 301 redirect instead of 302 for testing	Google treats the variant URL as canonical, replaces the original URL
10	Marking variants with noindex during testing	Main page is completely blocked
11	Running tests for half a year	Treated as deception, penalized
12	Server overloaded and returns 503 long-term	URLs permanently removed from index
13	Soft 404 (200 response with error content)	Removed from index after identification
14	Blocking important JS/CSS with robots.txt	Rendering fails, indexing quality drops
15	Frequently submitting unchanged sitemap versions	Wastes crawl budget
16	When deleting AMP, directly emptying files without removing	invalid document → Google still serves old version
17	Mobile pages use URL fragments	Not indexed
18	Keyword-stuffed internal links(`<a> <a> <a> <a>`)	Hard for readers to distinguish, weak signal

Part 12 — Related wiki

Complete robots.txt / noindex rules: Complete Guide to Crawling and Indexing Control (robots.txt / noindex / X-Robots-Tag / data-nosnippet / rel)
canonical / redirect / site move: Complete Guide to Canonical / Redirects / Site Migration (including full HTTP status code table + 503 temporary outage playbook)
Complete sitemap: Complete Sitemap Guide (XML / RSS / Text + image / video / news / hreflang extensions + sitemap index)
JS rendering / SPA pitfalls: Full JavaScript SEO (basics + diagnostics + Dynamic Rendering + Lazy Loading)
structured data: Complete Structured Data Overview (framework + quick lookup for 34 rich result types + JSON-LD/Microdata/RDFa selection)(upcoming wiki)
Viewing Crawl Stats / Index Coverage in Search Console: How to use the Search Console + Analytics + Trends trio (traffic diagnostics / keyword research / Looker Studio)
SEO basics + 3 phases (crawl / index / serve): Google SEO basics + How Search works + Search Essentials
Helpful content + 14 ranking systems: Google ranking systems + Core/Spam Updates + business information establishment
Hacked / Security: Complete guide to security / Spam / Removal / search operators (malware + phishing + UGC + redacted + site:)

Last updated

2026-05-17 — Compiled based on 14 articles in Google Search Central's crawling-indexing series (latest from 2025-12 to 2026-02). The 2026-02 Googlebot documentation covers AVIF image format + 64MB PDF limit / complete mobile-first troubleshooting / 14 valid robots rules including max-snippet usage for AI Overviews / full AMP lifecycle.