GSC Blog: Evolution of Googlebot / JS-SEO / Mobile-first / Page Experience (2016–2026)

TL;DR (one sentence): Over the decade from 2016 to 2026, Google Search did four major things — ① It replaced Googlebot from a "fake iPhone Safari" to a "Chrome-based, evergreen, mobile-default" crawler; ② It shifted indexing from "desktop-primary" to "mobile-only" (announced in 2016 → completed across the entire web in 2023); ③ It evolved JS rendering from "let sites prerender themselves" to "Googlebot rendering pages itself using evergreen Chromium, with dynamic rendering phased out from a stopgap solution to no longer being the first choice"; ④ It shifted Page Experience from "aggressively pushing AMP → soft CWV prompts → retired in 2023 to the position that 'there is no single page experience signal'." AMP is the part of this whole story that rose the highest and fell the hardest of this whole story.

Core principles (four unchanging laws that run through the entire decade):

Do not UA-sniff Googlebot — that is cloaking; feature detection + progressive enhancement is the correct approach.

Mobile-first indexing and mobile-friendliness are two different things — the former only concerns "which version is crawled" and gives no ranking advantage; the latter actually affects rankings.

Crawling is not a ranking signal — being crawled more ≠ ranking better; it is only the ticket to entry.

There is no single "page experience signal" — Core Web Vitals / HTTPS / mobile-friendliness / no intrusive interstitials are just a few among "hundreds of signals", and search intent > page experience (slow pages can still rank high).

Part 1 — Evolution of Googlebot: From desktop to evergreen to mobile-default

1.1 Full timeline

Time	Event	Key details
2016-04-18	Smartphone Googlebot switched from a fake iPhone to an Android Nexus 5X	UA changed to `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) ... Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1)`. Reason: "The renderer is closer to Chrome than to Safari"
2019-05-07	The New Evergreen Googlebot — Rendering engine jumped straight from Chrome 41 to the latest stable Chromium 74	Announced by Martin Splitt. Unlocked over 1,000 new web platform features at once: ES6+, IntersectionObserver, Web Components v1
2019-08-07	Testing tools also upgraded to evergreen in lockstep	URL Inspection / Mobile-Friendly Test / Rich Results Test / AMP Test all aligned,rendering side upgraded first, UA string not yet changed
2019-10-02	Announcement that starting in December a `Chrome/W.X.Y.Z` placeholder would be added to the UA string — rolling in step with Chrome version numbers	Desktop/mobile UAs updated synchronously;evergreen status officially visible at the UA layer
2020-09-17	Googlebot will soon speak HTTP/2	Starting in November, h2 enabled for some sites;Only HTTPS sites that support h2 are included, no h2c. Opt out using the `421` status code
2022-06-28	Clarified that "Googlebot has a 15 MB upper limit for fetching a single resource"	This was not a new limit, it was simply "documented for the first time".Each subresource (CSS/JS) has its own 15 MB upper limit(2023-03-16 update); any portion of HTML beyond 15 MB is dropped
2023-10-31	Desktop Googlebot essentially retired	Only retained as a fallback crawler for a small number of broken mobile sites; Search Console removed display of indexing crawler information

1.2 Google’s own words (core advice repeated over many years)

(2016-03) "User agent sniffing for Googlebot is not recommended and is considered to be a form of cloaking. Googlebot should be treated like any other browser."
(2019-10) "Sites that follow our recommendations to use feature detection and progressive enhancement instead of user agent sniffing should continue to work without any changes."
(2020-09) "Whether we crawl using h1 or h2 does not affect how your site is indexed... Is there any ranking benefit for a site in being crawled over h2? No."

1.3 Recurring pitfall patterns

Returning errors to the bot after UA sniffing — for example, blocking Googlebot as if it were "a user with an ad-blocker"; typical symptom: what Googlebot crawls is not the normal page but an error message.
The mobile version uses noindex/nofollow but the desktop version does not — after the mobile-first switch, indexing can suddenly be lost.
Blocking .css / image URLs — Googlebot cannot fully render, leading to ranking drops.
Making key SEO judgments on content beyond the 15 MB limit — it is all dropped.Be sure to extract inline scripts / CSS dust to external files(original quote from June 2022).

Part 2 — The full transition history of Mobile-first indexing (a 7-year long march)

2.1 Full timeline

Time	Key milestone	Original quote cited
2016-11-04	Mobile-first indexing (Doantam Phan) — First official announcement, experiments begin	"begun experiments to make our index mobile-first... we will continue to be a single index"
2018-03-26	Rolling out mobile-first indexing — After 1.5 years of experimentation, site-by-site migration began, with notifications sent via Search Console	"Mobile-first indexing is about how we gather content, not about how content is ranked. No ranking advantage."
2018-12-19	MFI + structured data + images — More than half of global results had switched over	Two classic pitfalls: ① missing structured data on mobile; ② missing alt text on mobile. "A page does not need to be mobile-friendly in order to be considered for mobile-first indexing."
2019-05-28	Mobile-First Indexing by default for new domains — New domains have MFI enabled by default starting 2019-07-01	"default state for new websites will be mobile-first indexing, there's no need to send a notification"
2020-03-05	Announcing MFI for the whole web — 70% had switched over,full-web deadline set for 2020-09	Responsive design recommended,m-dot not recommended:"because of issues and confusion we've seen from separate mobile URLs"
2020-07-22	Prepare for MFI (with extra time) — Postponed to 2021-03 due to COVID	Added 5 new categories of pitfalls: do not tie lazy-loading to user interaction / do not block CSS / mobile headings must not be watered down / do not use overly small thumbnails for images / video placement must be preserved
2023-10-31	Mobile-first is here — End of the 7-year long march	Desktop Googlebot essentially retired, only retained as a fallback for a tiny number of broken mobile sites

2.2 Unchanging core principles

One single index, not two separate sets — "We continue to have one single index that we use for serving search results. We do not have a 'mobile-first index' that's separate." (reiterated repeatedly starting 2018-03)
Content must be equivalent: text / images / videos / links / structured data / meta tags / titles / descriptions / robots meta — the mobile version and desktop version must all be on an equal footing, otherwise traffic will be lost when MFI switches over.
MFI ≠ mobile-friendly: MFI is about "which version is crawled", while mobile-friendliness is about "whether the page works well on small screens"; the two are orthogonal.
Responsive design recommended, m-dot not recommended: Starting in 2019, Google officially stated it does not recommend separate mobile URLs.

2.3 Inspection tools

Search Console URL Inspection Tool — Check the "Crawled as" field.
Settings page — Shows the current site status.
Server logs — Check whether the share of Smartphone Googlebot crawls is rising.

Part 3 — JavaScript SEO: From "you prerender" to "I render"

3.1 The full evolutionary arc

Time	Milestone	Position
2017-12-04	Rendering AJAX-crawling pages (John Mueller) — Announced that the AJAX crawling scheme (`#!` + `_escaped_fragment_=`) would be retired in 2018 Q2	"we'll no longer be using the AJAX crawling scheme... Googlebot will render the `#!` URL directly" — First explicit statement that "I will render it myself"
2019-01-31	Dynamic Rendering with Rendertron (Martin Splitt) — A transitional solution before evergreen	dynamic rendering = Serve prerendered HTML to bots based on UA switching, client-side render to users; Rendertron is a headless Chromium implementation
2019-03-06	Introducing a new JavaScript SEO video series (Martin Splitt) — YouTube series launched	Covering: classic vs JS sites / how Googlebot crawls-renders-indexes / React Angular Vue basics / testing and debugging tools / dynamic rendering with Rendertron
2019-05-07	The New Evergreen Googlebot — Chromium 74 upgrade, 1000+ new features	Dynamic rendering effectively became a "fallback workaround" rather than the "first choice" — Googlebot itself can run ES6+
2019-08-07	Testing tools also upgraded to evergreen	Webmasters can directly see Googlebot’s rendered output in the tools

3.2 Current position (based on post-2019 messaging)

First choice: Googlebot itself renders your JS using evergreen Chromium — you do not need to do any special handling.
Second choice: hybrid rendering (SSR + client-side hydration, such as Angular Universal) — improves both user FMP and crawler visibility.
Last resort: dynamic rendering (Rendertron / Puppeteer / Rendora, etc.) — "useful for content that changes often and needs JavaScript to display", but it is a workaround. Later (after 2022) Google documentation effectively labeled dynamic rendering as "not recommended as a long-term solution"; this batch of 38 posts does not directly cover that deprecation announcement, but the evergreen upgrade effectively handed it a suspended death sentence.
Never do: return different content to Googlebot versus users (that is cloaking). The prerendered HTML output by Rendertron must be equivalent to what the user side sees after hydration.

3.3 Core viewpoints from Martin Splitt’s series (refined from the starting point of the 2019 series)

Challenges JavaScript brings to SEO:

Rendering takes time and resources — Googlebot may delay indexing your content.
Need to distinguish clearly: initial HTML, rendered DOM, indexed snapshot.
React, Angular, and Vue each have their own SSR solutions; it is recommended to prioritize the SSR/SSG built into the framework.

Part 4 — The Rise and Fall of Page Experience / Core Web Vitals Signals

4.1 Complete Timeline

Time	Milestone	Key Changes
2018-01-17	Using page speed in mobile search ranking — "Speed Update"	Launched in 2018-07; "only affect pages that deliver the slowest experience";"a slow page may still rank highly if it has great, relevant content"
2018-01-10	Real-world data in PageSpeed Insights	PSI integrated CrUX real-user data; Speed Score (FCP+DCL) / Optimization Score / distribution chart
2018-11-12	PageSpeed Insights now powered by Lighthouse	Lighthouse became the unified engine; PSI = Lab Data (Lighthouse simulation) + Field Data (CrUX 28-day data)
2020-05-28	Evaluating page experience — First announcement that Core Web Vitals would become a ranking signal, and previewed the removal of the hard AMP requirement for Top Stories	Signals = CWV (LCP+FID+CLS) + mobile-friendliness + HTTPS + no intrusive interstitials. "A good page experience doesn't override having great, relevant content." Delayed due to COVID,with at least 6 months of advance notice provided
2020-11-10	Timing for page experience	Confirmed launch in 2021-05; planned to test "adding a visual indicator in search results to mark pages with good experience"
2021-04-19	More details on page experience update	Actual gradual rollout started on 2021-06-15 and completed by the end of August; Top Stories officially opened to non-AMP content;The AMP badge icon was removed from Top Stories; SXG opened to all content(previously restricted to AMP only)
2021-08-04	Simplifying the Page Experience report	Safe Browsing was explicitly clarified as not a ranking signal, and removed from the PE report; the Ad Experience widget was also removed (it was never a PE factor); updated PE signal diagram: LCP/FID/CLS + Mobile Friendliness + HTTPS + No Intrusive Interstitials (6 items total)
2021-11-04	Bringing page experience to desktop	Starting 2022-02, desktop also uses CWV; the mobile-friendliness signal does not apply to desktop
2023-04-19	The role of page experience in creating helpful content — Effectively retired	"There is no single signal. Our core ranking systems look at a variety of signals that align with overall page experience." The Search Console Page Experience report was changed to only link to documentation;Starting 2023-12-01, the Mobile Usability report, Mobile-Friendly Test tool, and related API were all retired
2024-03-12	INP officially replaced FID in CWV (marked in update paragraphs across all PE articles)	—

4.2 Key Quotes Connected in Sequence

(2018-01) "A slow page may still rank highly if it has great, relevant content."
(2020-05) "A good page experience doesn't override having great, relevant content. However, in cases where there are multiple pages that have similar content, page experience becomes much more important for visibility in Search."
(2021-04) Used a metaphor for the PE update as adding seasoning: "Rather than add the flavor all at once into the mix, we'll be slowly adding it all over this time period."
(2023-04) "There is no single signal. Our core ranking systems look at a variety of signals that align with overall page experience."
(2023-04) "Page experience is not an eligibility requirement to appear anywhere in the 'Top stories' section."

4.3 The Full Path of PE Signals "Softening"

Strong announcement in 2020 → 2021 delay + gradual rollout + removal of AMP requirement + removal of Safe Browsing → 2023 explicit statement of "no single signal" + retirement of dedicated report + retirement of Mobile Usability report + retirement of Mobile-Friendly Test.

Conclusion: Page Experience softened from a "standalone ranking signal" in 2020 to "one component of helpful content, with no single calculation formula" in 2023. CWV (especially LCP/INP/CLS) remains important, but it is not a guarantee of ranking.

Part 5 — The Rise and Fall of AMP (A Full Decade-Long Saga)

5.1 Beginning — Aggressive Push Period 2015-2016

2015-10 (Citation) The AMP Project launched as open source.
2016-01 AMP error report in Search Console — A preview shown first to publishers.
2016-03 AMP NewsLab Office Hours — Promoted in 8 languages(French/Italian/German/Spanish/Portuguese/Russian/Japanese/Indonesian), demonstrating global rollout intensity.
2016-08 AMP your content — Expanded from Top Stories to the entire mobile search results page. 150M AMP docs indexed, with 4M added per week150M AMP docs / 4M added weekly. WaPo reported that "AMP reduced load times by 88% and increased returning mobile search users by 23%".
2016-09 Published 3 posts in one week (What is AMP / Webmaster forums Top Q / 8 tips to AMPlify); maintained the official line: "AMP is not a ranking signal".
2016-10 Search Console launched a standalone AMP testing tool that supported sharing snapshots.

5.2 Peak and Boundaries — 2017-2019

2017-11 Engaging users through high-quality AMP — Content parity policy(Mandatory starting 2018-02-01): AMP content must be equivalent to canonical content; teaser AMP pages will be demoted out of Top Stories (but this does not affect core search rankings themselves). 25M domains had adopted AMP.
2019-04 Instant-loading AMP from your own domain — Signed Exchanges (SXG) Launched, resolving the biggest user complaint that "the address bar shows google.com/amp instead of the publisher's domain".

5.3 Retreat — 2020-2023

2020-05 Evaluating page experience — AMP is no longer an entry ticket to Top Stories:"AMP will no longer be necessary for stories to be featured in Top Stories on mobile; it will be open to any page."
2020-08 Identify and fix AMP SXG errors — Search Console added an SXG error report (a late-stage remediation tool).
2020-11 Timing for page experience — AMP "one of the easiest and cost-effective ways" to achieve good PE (offering AMP a graceful step down).
2021-04 More details on page experience update — The AMP badge icon was removed from Top Stories; SXG is no longer AMP-only, and opened to all content (this was another demerit: SXG was once an exclusive technical advantage of AMP).
2023-04 Page experience in search — The Page Experience report no longer emphasizes AMP;the AMP era effectively ended. There is no standalone "AMP deprecated" announcement across these 38 posts, but official mentions of AMP dropped from several per month on average in 2016 to nearly zero in 2023.

5.4 Boundaries That Never Changed

"AMP is not a ranking signal." — Remained word-for-word unchanged from the first post in 2016 through 2023. Only "secondary privileges" such as Top Stories entry, address bar branding, and SXG exclusivity were removed one after another, completely eliminating adoption incentives.

5.5 Recurring Lessons

content parity — Taught a lesson to a group of publishers that built teaser AMP pages.
AMP cache + signed exchange resolved URL display issues — but implementation was complex, requiring CDN/server coordination.
"AMP is the easiest way to achieve good PE" was Google's final graceful out for AMP — but since PE already has "no single signal", this justification no longer holds.

Part 6 — Official Explanation of Crawl Budget

6.1 The Only Authoritative Definition

2017-01-16, Gary Illyes — The only dedicated crawl budget post among these 38 articles, and remained the canonical answer for many years afterward.

Crawl budget = Crawl rate limit × Crawl demand

Crawl rate limit: number of concurrent connections + fetch interval; determined by crawl health(site speed + 5xx rate) and Search Console settings together.

Crawl demand: determined by popularity(URL popularity) and staleness(preventing index staleness).

6.2 The Two Most Critical Boundaries

"If new pages tend to be crawled the same day they're published, crawl budget is not something webmasters need to focus on. Sites with fewer than a few thousand URLs have no need to worry at all."
"Crawling is NOT a ranking signal." (Increased crawl rate ≠ better positions)

6.3 6 Types of Low-Value URLs That Drain Crawl Budget (Ranked by Importance)

Faceted navigation + session identifiers
On-site duplicate content
Soft error pages
Hacked pages
Infinite spaces + proxies
Low quality + spam

6.4 Q&A Quick Reference

Question	Answer
`crawl-delay` robots rule	Googlebot does not process
`nofollow` Can that save crawl budget	Not necessarily — other pages or external sites may link to it without nofollow, so it will still be crawled
`robots.txt disallow` Does it count toward crawl budget	It does not count
Do AMP / hreflang / CSS / JS / AJAX count toward crawl budget	They all count
Long redirect chains	hinder crawling
Does site speed affect crawl budget	Yes, it does — fast sites = healthy = given more budget; 5xx errors + timeouts → reduced budget

Part 7 — Miscellaneous Topic Fragments

7.1 SEO Starter Guide Revision (2017-12)

deprecate: Webmaster Academy + old SEO Starter Guide PDF
Replaced by: new SEO Starter Guide, adding three major sections on the necessity of SEO / structured data / mobile-friendliness
Launched in 9 languages, later expanded to 25 languages

7.2 Crawling December 2024 Series

The candidate list of these 38 posts does not include the Crawling December 2024 series(caching / CDNs / faceted nav / resources). If the user adds those 4 posts later, this Part will be expanded. It can currently be linked as further reading for Part 6 crawl budget.

7.3 Evolution of the PageSpeed Insights Tool Itself (Supporting the Main Thread)

2018-01 Introduced CrUX real-user data → Speed Score (FCP+DCL)
2018-11 Switched to Lighthouse unified engine → dual Lab + Field data
2018-07 Spanish translation announcement (simultaneous multilingual promotion, no substantive content)

Quick Reference Cheatsheet for Myself/My Team

7 High-Frequency Questions Encountered in SEO → Answers:

Should I implement AMP? — 2026 stance:No. Top Stories is already open, the AMP badge has been removed, and SXG has been generalized. Unless you have already invested heavily in AMP infrastructure, new projects should not adopt AMP.
Should I implement dynamic rendering? — Do not use it as a long-term solution. Googlebot now runs evergreen Chromium, so direct CSR is fine; if both FMP and SEO are tight, prioritize SSR/SSG.
Do I need to worry about crawl budget? — Sites with fewer than a few thousand URLs do not need to think about it at all. Large sites should first fix faceted navigation and duplicate content.
Where did the mobile-friendly test tool go? — Retired on 2023-12-01. Use Lighthouse or PageSpeed Insights instead.
Are Core Web Vitals still important? — Important, but they do not guarantee rankings. Good CWV acts as a tiebreaker in scenarios with "multiple relevant pages"; content relevance always comes first.
How do I identify the Googlebot UA? — Always use feature detection, do not sniff UA; if you must sniff, match the "Googlebot" substring and verify with reverse DNS.
What if the page is larger than 15MB? — Extract inline scripts/CSS to external files; data URIs count toward the HTML size; each subresource has a 15MB upper limit.

History

2026-05-17 v1.0 — Read through all 38 GSC blog posts in one go and distilled 7 major themes.Used a two-phase strategy: "read 15 posts first and commit v0.1 + append after finishing reading for v1.0"To prevent stream timeouts. All Parts are complete; the Crawling December 2024 series is only a placeholder because it falls outside the scope of the 38 posts.