GSC Blog: Robots / Sitemap / Canonical / Hreflang / Indexing API(2016-2026)

A thematic distillation of 15 Google Search Central blog posts spanning 11 years (2018–2025). It covers: ① the full process of robots.txt evolving from "Google's internal rules" to the "IETF RFC 9309 international standard"; ② the continuous simplification of the Sitemaps protocol (ping, extension tags, changefreq, and priority all deprecated); ③ expansion of page-level controls (meta tag + X-Robots-Tag + indexifembedded); ④ nofollow changing from a hard rule to a hint, plus the introduction of sponsored/ugc; ⑤ the strict two-category boundary of the Indexing API (only job posting and livestream).

Three cross-post main themes: (1) "If it's not supported, it's not supported" — Google publicly acknowledged that the never-documented noindex/crawl-delay/nofollow does not work in robots.txt, and removed it from source code on 2019-09-01; (2) "The ping endpoint is completely dead" — sitemap ping, sitelinks search box, and most extension tags for image/video sitemaps were centrally deprecated between 2022 and 2024, reflecting Google's preference for cutting low signal-to-noise ratio signals outright; (3) "The Indexing API is not for indexing everything" — it only accepts two categories of short-lived URLs: job posting and livestream; there is no API entry point for other scenarios.

Part 1 — robots.txt Standardization: 2019 IETF + 2024 Refresher Series

1.1 Timeline (the single most important chart)

Year	Event	Source
1994	robots.txt is born (4 years before Google)	2025/02 intro-robots-refresher
2005	Sitemaps Protocol released	2023/06 sitemaps-lastmod-ping
2007	Major search engines uniformly support the `sitemap` directive in robots.txt	2025/02 intro-robots-refresher
2019-07-01	Google open-sources its production robots.txt parser (C++)	2019/07 unsupported-rules
2019-07-02	Google announcement: unsupported directives to be removed from code starting 9-01	2019/07 unsupported-rules
2019-09-01	`noindex` / `crawl-delay` / `nofollow` becomes completely ineffective in robots.txt	2019/07 unsupported-rules
2020-09	Intern contributes Java port + test framework	2020/09 open-source-projects
2022	robots.txt becomes IETF RFC 9309	2025/02 intro-robots-refresher
2025-02	Robots Refresher series opens	2025/02 intro-robots-refresher
2025-03	Three Refresher posts released in succession (flexible / page-level / future-proof)	Three posts in 2025/03

1.2 Key facts: these directives in robots.txt were never supported by Google

Original wording from 2019/07: “Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low. Digging further, we saw their usage was contradicted by other rules in all but 0.001% of all robots.txt files on the internet.”

Removed / unsupported list:

noindex(in robots.txt) — Never documented by Google; code removed starting 2019-09-01
crawl-delay — Not supported by Google, but may be supported by other engines
nofollow(in robots.txt) — Same as above
clean-param — Mentioned in 2025/03 robots-future as "not in RFC 9309, but supported by some engines; not supported by Google Search"

Five correct alternatives to noindex (2019/07):

robots meta in the noindex ( tag is the most effective deindexing method, provided crawling is allowed)
HTTP 404 / 410
Password protection
robots.txt Disallow(If crawling is blocked, it is very unlikely to be indexed, but may still enter the index due to external links — it will just become increasingly invisible)
Search Console Remove URL tool (temporary removal)

1.3 Extensible "de facto standards" beyond RFC 9309

Original wording from 2025/03 robots-future: “The "sitemap" rule, which again is not part of RFC9309, is supported by all major search engines. Given enough support, it could become an official rule in the REP.”

How to make a new directive a standard (in Gary Illyes' words):

Discuss your idea publicly(unilateral implementation does not count as a standard)
Make sure both publishers and crawlers see benefits
Once consensus is reached → there is a possibility to update the RFC
Path reference:sitemap directive gained bottom-up consensus in 2007

1.4 Core arguments of the Robots Refresher series (three posts in 2025)

Post	Core argument
2025/02 intro	robots.txt is older than Google (1994 vs 1998), an "essential tool that enables the internet to work"; it only entered the IETF in 2022 but had already been in effect for 28 years
2025/03 flexible	robots.txt is a "Swiss Army knife"; 5 common patterns are given (shared rules for multiple bots / paths containing specific strings / allow + disallow combinations / blocking AI trainer bots separately / `#` comments)
2025/03 page-level	robots `meta` The `X-Robots-Tag` tag was proposed by the W3C in 1996 (before Google), later extended to the HTTP header; outside HTML (PDFs/images/documents), the header must be used
2025/03 future	REP can "grow up", but the threshold must be high; public discussion + bilateral consensus + long-term use = the path to standardization

1.5 Choosing robots.txt vs page-level: when to use which (2025/03 page-level)

Scenario	Tool
Want to block the "crawl" behavior itself (e.g., infinite search result pages)	Only robots.txt
FTP server control	Only robots.txt
Don't want snippets to appear	Only page-level (`nosnippet`)
No distinction between crawl vs index, and broad blocking is needed	robots.txt (coarse granularity)
Single-page control	page-level meta tag / X-Robots-Tag

1.6 Official syntax for blocking AI training bots (example given in 2025/03 flexible)

user-agent: *
allow: /

user-agent: aicorp-trainer-bot
disallow: /
allow: /$

— Meaning: except for the homepage, aicorp-trainer-bot cannot crawl any path.

1.7 Open-source robots.txt ecosystem (2020/09)

C++ parser(open-sourced 2019) — identical to the one used in Google's production systems
Java port(2020/09) — Google's internal team "already plans to use it in production systems"
robots.txt-spec-test(2020/09) — Test framework by intern Andreea Dutulescu, which verifies whether any parser conforms to REP

Part 2 — Evolution of Sitemap Best Practices

2.1 The "continuous simplification" of the Sitemaps protocol

Year	Deprecated item	Current status
2022-05	Image sitemap: `caption` / `geo_location` / `title` / `license`	Ineffective after 2022-08-06; use alt text + IPTC metadata instead
2022-05	Video sitemap: `category` / `player_loc[@allow_embed]` / `player_loc[@autoplay]` / `gallery_loc` / `price[@all]` / `tvshow[@all]`	Same as above
2023-06	Sitemap ping endpoint	Returns 404 after 6 months; "vast majority of submissions lead to spam" (Bing deprecated it simultaneously)
Continues	`changefreq`	Google does not use it at all, and it overlaps conceptually with `lastmod` Concept overlap
Continues	`priority`	Google does not use it at all; "based on internal studies, it generally doesn't accurately reflect actual priority"

2.2 `lastmod` is the only element in sitemaps that Google takes seriously

Original wording from 2023/06: “lastmod is indeed useful in many cases and we're using it as a signal for scheduling crawls to URLs that we previously discovered.”

lastmod Four correct usage rules for

Format must be valid(per sitemaps.org documentation; Search Console will report errors)
Must match reality — “if your page changed 7 years ago, but you're telling us in the lastmod element that it changed yesterday, eventually we're not going to believe you anymore”
You can fill it in only for pages you are sure about — for aggregate pages like homepage / category pages where the real modification time is unknown, it is better not to fill it in
Must be a "significant modification" — “last significant modification"; changing the sidebar/footer does not count; update only when main content/structured data/links are changed

2.3 Sitemap submission methods (only two ways after 2023-06)

Add the Sitemap: directive in robots.txt
Manual submission in Search Console

Cut: unauthenticated REST ping endpoint (http://www.google.com/ping?sitemap=...). No code changes needed; calls simply return 404.

2.4 A strange but working-as-intended pitfall

xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" uses HTTP, not HTTPS.

Original wording from 2023/06: “This is working as intended: it's a reference for parsers about the elements in the XML. Please don't file more documentation feedback about this.”

Part 3 — Five Major Categories of Common rel=canonical Errors

Note: This batch of 15 blog posts does not include a dedicated post on canonical errors (the dedicated topic is in the sister wiki gsc-canonical-and-migration.md), but multiple posts indirectly touch on canonical issues; relevant first-hand discussions are aggregated here.

3.1 Priority of canonical / noindex / indexing signals (cross-referenced across multiple posts)

Signal	Strength	Invalidation condition
404 / 410	Strongest deindexing	Takes effect only when crawled
password protection	Strong	Does not depend on crawling
robots `meta` noindex	Strong	The page must be allowed to be crawled(meta cannot be seen when blocked by robots.txt)
X-Robots-Tag: noindex (HTTP header)	Same as above	Same as above; applies to PDFs/images
robots.txt Disallow	Weak deindexing (only prevents crawling)	When there are many external links, the URL may still enter the index but without a snippet
robots.txt `noindex`(deprecated)	No longer effective	Not recognized by code after 2019-09-01

3.2 Easy to confuse: blocked by robots.txt ≠ not indexed

Original wording from 2021/01 Index Coverage Data Improvements: “Pages that were submitted but blocked by robots.txt and got indexed are now reported as 'indexed but blocked' (warning) instead of 'submitted but blocked' (error).”

— Google itself classifies this as a "warning" rather than an "error" in Search Console, acknowledging that pages blocked by robots.txt may still enter the index(based on external links), even though the content cannot be read.

3.3 `indexifembedded`: Indexing refinement for new scenarios (2022/01)

Scenario: A media publisher wants iframes embedded on third-party pages to be indexed, but does not want the original playback page itself to be indexed.

Before 2022/01: A standalone noindex would cause Google to also fail to read iframe content when indexing third-party pages.

New solution:

<meta name="googlebot" content="noindex,indexifembedded">

or HTTP header:

X-Robots-Tag: googlebot:noindex,indexifembedded

Note:Only Google supports indexifembedded.

3.4 "Indexed without content" — new diagnostic category (2021/01)

Index Coverage report added indexed without content(warning) — the page is indexed but Google cannot read its content (common causes: rendering failure / JS did not run correctly / cloaking).

3.5 Soft 404 reporting made more accurate + the crawl anomaly catch-all removed (2021/01)

Removed generic "crawl anomaly" — all issues broken down into specific categories
Soft 404 reporting accuracy improved

Original wording from 2021/01: “all crawls errors should now be mapped to an issue with a finer resolution.”

Part 4 — Hreflang in Practice

Important note: This batch of 15 blog posts does not specifically cover hreflang(dedicated hreflang articles were published earlier in 2014/2018 and have been placed in the sister wiki gsc-international-and-explicit.md). This section only records indirectly relevant information from these 15 posts for cross-reference.

4.1 Diagnostic entry point for multi-region / multi-language crawling

The "domain properties + multiple hosts" support in the 2020/11 Crawl Stats Report is the entry point for checking the crawl health of cross-language subdomains:

2020/11 original quote: “For domain properties with multiple hosts, you can check the host status for each of the top hosts presented in the report summary view. This can help you evaluate performance of all hosts under your domain in one place.”

— Common subdomains for cross-language/regional setups (fr.example.com / de.example.com), you need to view the host status for each host from the domain property perspective.

4.2 For details on hreflang, see

→ wiki/gsc-international-and-explicit.md → wiki/gsc-canonical-and-migration.md

Part 5 — Indexing API (boundaries for job posting / livestream)

5.1 Strict boundary: The Indexing API only accepts two types of content

2018/06 original quote: “Currently, the Indexing API can only be used for job posting pages that include job posting structured data.”

2018/12 original quote: “The Indexing API now supports pages with livestream structured data.”

Status 5 years later (as of this batch of documents): It still only supports these two types.There is no general-purpose Indexing API that lets you push arbitrary URLs to Google.

5.2 Why only these two types

Common trait: Short-lifecycle URLs (job postings are taken down once filled / livestreams have a fixed start-end time window)
Pain point: Crawl schedules cannot keep up → users land on expired content
Solution: Site owners actively send push notifications so Google can schedule a fresh crawl

2018/06 original quote: “For websites with many short-lived pages like job postings, the Indexing API keeps job postings fresh in Search results because it allows updates to be pushed individually.”

5.3 When to call the API

Job posting: Call when published + call when removed (two endpoints)
Livestream: Call when it starts + Call when it ends + Call when structured data changes(clarified in 2018/12)

5.4 Prerequisites

The page must contain the corresponding structured data (JobPosting / VideoObject + livestream guidelines)
otherwise calling the API is useless

5.5 Implicit red line

— Using the Indexing API on any page that is not a job posting or livestream counts as "abuse", and will be rate-limited in the long run. Google has not stated this explicitly, but it is a consensus from years of community practice.

Part 6 — Evolution of noindex / nofollow

6.1 Key timeline point when nofollow changed from a "hard rule" to a "hint" (2019/09 + cross-article references)

Phase	Time	Semantics of nofollow
2005-2019	nofollow introduced to combat comment spam	Hard rule: Marked links are not counted as ranking signals
Starting 2019-09-10	Introduced `sponsored` / `ugc` two new rel values	nofollow becomes hint(for ranking scenarios)
Starting 2020-03-01	At the crawling/indexing layer	nofollow also becomes hint(no longer treated as a hard rule)

6.2 Semantic boundaries of the three rel values (2019/09)

rel value	Purpose
`rel="sponsored"`	Links created from advertising / sponsorship / compensation agreements
`rel="ugc"`	Links in user-generated content (comments / forums)
`rel="nofollow"`	Do not want to imply any endorsement (fallback)

6.3 Core FAQ conclusions (2019/09)

Existing nofollow markup does not need to be changed
Multiple rel values can be combined:rel="nofollow ugc" / rel="ugc sponsored"
Should ad links be marked?They must be marked(use sponsored or nofollow, otherwise you risk a link scheme penalty)
What if you mark them incorrectly?Only marking a non-ad link as "sponsored" is a directional error — Marking a non-ad link as sponsored results in "no endorsement credit counted", with no major harm; conversely, failing to mark ads with sponsored/nofollow is the real major problem
Why change from "ignore" to "hint"? — "Links contain valuable information that can help us improve search, such as how the words within links describe content they point at." (Even nofollow links have signal value from their anchor text)

6.4 Evolution of noindex

Time	Event
History	noindex in robots.txt was never supported by Google, but many people misused it
2019-09-01	Code that handled noindex in robots.txt was completely removed
2022-01	Introduced `indexifembedded`, allowing noindex to be treated with leniency in some cases
2025-03	The page-level granularity article explicitly states: noindex was proposed in 1996, older than Google itself

6.5 Obsolete values like noodp (2025/03 page-level)

Original quote: “sometimes values are deprecated, as was the case with noodp, which used snippets from DMOZ / Open Directory Project before the directory was closed.”

— Reminder: The list of supported values for robots meta is not static; values may be added or removed.

Although it does not directly belong to the 6 major topics, it reflects Google's consistent preference for cutting "low-usage features":

Introduced 2014/09
Fully shut down on 2024/11/21
Effective globally, does not affect rankings
Structured data can be retained and will not trigger errors
WebSite structured data is still supported(site names use the same schema variant)

Appendix B — Evolution of the Index Coverage Report (2021/01)

Old issue	New issue	Type changes
crawl anomaly (catch-all bucket)	was broken down into specific issues	—
submitted but blocked (error)	indexed but blocked (warning)	error → warning
—	indexed without content	New warning added
Soft 404 reporting	Accuracy improved	—

Appendix C — Core new capabilities of Crawl Stats Report v2 (2020/11)

Total requests by response code / file type / crawl purpose / Googlebot type
Host status detailed information (90-day availability)
URL examples (where requests were made)
Domain properties Multi-host consolidated view

Compilation of key quotes across articles

“[the unsupported rules' usage] was contradicted by other rules in all but 0.001% of all robots.txt files on the internet”(2019/07) — Official reason for removing noindex in robots.txt
“the vast majority of the submissions lead to spam”(2023/06) — Official reason for removing sitemap ping
“if your page changed 7 years ago, but you're telling us in the lastmod element that it changed yesterday, eventually we're not going to believe you anymore”(2023/06) — lastmod trust mechanism
“Links contain valuable information... By shifting to a hint model, we no longer lose this important information”(2019/09) — Design reason for changing nofollow to a hint
“robots.txt is here to stay... it's easy, it's granular and expressive, it's well-understood and accepted, and it just works, like it's been working for decades now.”(2025/02) — Google's long-term commitment to robots.txt
“The most important thing you, the reader, can do is to talk about your idea publicly and gather supporters for that idea”(2025/03 future) — REP extension mechanism
“Currently, the Indexing API can only be used for job posting pages”(2018/06) — Indexing API boundaries

Author distribution

Gary Illyes ×6 (2019/07 unsupported, 2019/09 nofollow co-author, 2020/09 open-source co-author, 2022/01 indexifembedded co-author, 2022/05 sitemap-cleaning, 2023/06 lastmod-ping, 2025/03 future)
John Mueller ×3 (2024/10 sitelinks-searchbox, 2025/02 intro-refresher, 2025/03 page-level co-author, 2025/03 flexible co-author)
Martin Splitt ×1 (2025/03 flexible co-author)
Danny Sullivan ×1 (2019/09 nofollow co-author)
Others: Zach Clifford / Danielle Marshak / Edu Pereda / Hillel Maoz / Tal Yadid / Weizi Wang, 1 entry each