36Kr Pitchhub Scraping Methodology

2026-05-19 field practice: went from "WebFetch timeout" to a complete pipeline in 1 hour, successfully running the three stages of 1135 + 2530 + 9448. 5-phase scripts are at ~/kb/sources/reference/36kr-pitchhub-2026/_scraper/,resumable.

Core Principles

The standard three-step approach for scraping data from SPA sites:

1. Look for SSR injection (if list page is empty → skip; if detail page is fat → directly curl and parse window.__*__)
2. Look for JSON API (when there is no SSR, reverse the bundle.web.js calls inside fetch("gateway", "/path"), inspect the request wrapper functions in the chunks)
3. Look for cookie / signature (if filter returns NPE or 422 = missing wrapper fields or cookie session)

Anti-scraping measures are not necessarily uniform across the entire site. The biggest pitfall in this case:36kr.com/p/<id> triggers captcha,but www.36kr.com/p/<id> is completely open. For the exact same content, adding a www. subdomain lets it pass.

5 steps of API reverse engineering

Step 1: Check if the main HTML contains SSR data

curl -sL -A "Mozilla/5.0" "https://pitchhub.36kr.com/investevent?financingTimeList[0]=2026" | \
  grep -oE 'window\.__[A-Z_]+__\s*=' | sort -u

Output injection points such as window.__INIT_PROPS__, window.__INIT_STORE__.Extract complete JSON using bracket matching(regex cannot be used, because nested curly braces inside JSON will break the match):

def extract_init_props(html):
    marker = "window.__INIT_PROPS__ = "
    start = html.find(marker) + len(marker)
    depth = 0; in_str = False; esc = False
    for i in range(start, len(html)):
        c = html[i]
        if esc: esc = False; continue
        if c == "\\": esc = True; continue
        if c == '"': in_str = not in_str; continue
        if in_str: continue
        if c == "{": depth += 1
        elif c == "}":
            depth -= 1
            if depth == 0: return json.loads(html[start:i+1])

List page:__INIT_PROPS__ only contains layout configuration, no data → SPA + asynchronous fetch Detail page:__INIT_PROPS__ contains the full projectData / krReportData / mentionData / industryData / projectRecomData dataset ← direct curl is sufficient

Step 2: Reverse engineer JS bundles to find API paths

Download the chunks referenced by the page:

curl -sL "https://static.36krcdn.com/36kr-pitchhub/web/bundle.web.<hash>.js" -o bundle.js
curl -sL "https://static.36krcdn.com/36kr-pitchhub/web/entrypoints_financingLibrary_Pc.web.<hash>.js" -o finpc.js

Grep for fetch("gateway"-style calls inside the chunks:

grep -oE '"/[a-zA-Z][a-zA-Z0-9_/.-]{4,80}"' finpc.js | sort -u | grep -iE 'invest|financ|project'
# 输出:
# "/pms/project/financing/list"          ← 这就是列表 API
# "/pms/investor/project/detail"

More precise: find the class definition class extends r.a { request() { return this.fetch("gateway","/pms/project/financing/list").post(this.params)... }} — it contains path + method + parameters.

Step 3: Reverse engineer the request wrapper format

First raw POST → 500 NPE. Need to find the wrapper in JS:

// 在 bundle.js 找到 partner_id 包装函数:
function u(t, r="deep", n="") {
  let i = {
    partner_id: n || "web",
    timestamp: (new Date).getTime(),
    partner_version: e.env.VERSION || "1.0.0",
    param: t,        // 用户参数嵌套在 param 字段
  };
  if (r === "deep") {
    if (!t.siteId) i.param.siteId = 1;
    if (!t.platformId) i.param.platformId = 2;
  }
  return i;
}

Send requests in this format,still NPE. The server-side SignFilter / partner validation passed, but the controller threw an internal NPE — it was unclear what was missing. At this point do not guess parameters blindly; proceed directly to Step 4 for live testing.

Step 4: Use Playwright to capture real requests for comparison

from playwright.sync_api import sync_playwright
events = []
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True, proxy={"server": "http://127.0.0.1:7890"})
    ctx = browser.new_context()
    page = ctx.new_page()
    def on_req(r):
        if "gateway.36kr.com" in r.url:
            events.append({"type":"REQ","url":r.url,"headers":dict(r.headers),"body":r.post_data})
    def on_resp(r):
        if "gateway.36kr.com" in r.url:
            events.append({"type":"RESP","url":r.url,"status":r.status,"body":r.text()[:2000]})
    page.on("request", on_req); page.on("response", on_resp)
    page.goto("https://pitchhub.36kr.com/investevent", wait_until="networkidle")

Compare the real request body against my manually constructed one → they were identical. The difference was in cookie: when Playwright visited the list page,Tencent SLB set SERVERID sticky cookie, and subsequent API calls went to the same gateway, so the controller's internal state was valid.

Solution: instead of manually forging cookies in curl (SERVERID has a short TTL), use Playwright to visit the list page once + use page.evaluate("fetch(..., credentials:'include')") so that fetch automatically carries the cookie.

Step 5: Bypass the hard 50-page limit (industry bucketing)

{"pageNo":"1","pageSize":20,"financingTimeList":["2026"],...}
# totalCount=1136 totalPage=50 pageSize=20
# pageNo>50 → "param.project.FinancingListReqForm.pageNo.Range"(422)
# pageSize>20 → "param.project.FinancingListReqForm.pageSize.Range"

The API has a hard limit of 1000 entries. Workaround:Bucket by industryIdList Bucketing, with each bucket containing < 1000 entries:

# 暴力探 industryId 1-27,每个都返非空即有效
for iid in range(1, 28):
    body = {"param":{"industryIdList":[iid], "financingTimeList":["2026"], ...}}
    # 各 industry totalCount: 1=19, 6=106, 9=210, 10=355(最大,但仍 <1000)...

Union of 26 valid industry buckets + overseas bucket(ifOverseas:1, 5 entries) → actually obtained 1135 / official count 1136 (1 entry lost due to unpacking error in industry 14).

Captcha subdomain differences (the biggest pitfall in this case)

URL	Results
https://36kr.com/p/<id>(bare root domain)	TTGCaptcha slider,even Playwright gets stuck (headless detection)
https://www.36kr.com/p/<id>	completely open, SSR window.initialState can be fetched directly
https://m.36kr.com/p/<id>	also open (article pages) — but /newsflashes/<id> triggers captcha
https://www.36kr.com/newsflashes/<id>	open (newsflash pages under www. path do not trigger captcha)

Do not rely on intuition for judgment; verify every path with curl. Content that triggers captcha:<script>window.TTGCaptcha.init(...)</script>, ByteDance lf-cdn-tos.bytescm.com/obj/static/sec_sdk_build/3.3.4/captcha/index.js.

Scraping architecture (hybrid Playwright + curl)

graph LR
  A[Playwright<br>open list] -->|SERVERID cookie<br>+ fetch evaluate| B[API list 1135 条]
  B --> C[curl<br>project detail<br>SSR fast]
  C --> D[curl<br>www. article<br>SSR fast]
  D --> E[curl<br>image binary<br>concurrent]
  classDef pw fill:#e3f2fd,stroke:#1976d2
  classDef cu fill:#f1f8e9,stroke:#558b2f
  class A pw
  class B,C,D,E cu

Playwright does only 1 thing: visit the list page for the first time, obtain the SERVERID cookie + call the list API. Everything else uses curl(8–16 parallel threads): because detail/article pages are all SSR, direct curl is 5–10 times faster than Playwright.

Phase pipeline:

Phase	Tool	Concurrency	Output
1: list	Playwright	Serial 50 API calls	_all_events.json 1000 entries
1b: Bucket-based gap filling	Playwright	Serial 27 industry calls	Fill 135 entries to reach 1135
2: detail	curl + ThreadPool	8	events/<pid>/detail.json
3: article	curl + ThreadPool	10	events/<pid>/articles/*.{md,json}
3b: Shared symlinks	Pure Python	—	280 cross-project symlinks
4: summarize	Pure Python	—	1041 summary.md + top-level INDEX
5: image	curl + ThreadPool	16	_images/<sha1>.<ext> 9448 images

Important details

Reverse-parsed detail.json fields

projectData is a nested dict; some fields are wrapped in {name, route} rather than being bare strings:

Field	Appears to be	Actually is
industryList[]	["先进制造"]	[{"name":"先进制造","route":"list_project?..."}]
currentFinancing	{round, money}	{"name":"B轮","route":"list_project?..."}(no money)
establishTime	"2024-08-21"	{"name":"2024年01月"}
business	string	dict (business registration data such as legalPersonName / regLocation / shareholder)
financingList[].vc	string	string (delimited by ", ") or [{name, id}] investor list

Use a generic names_of() to unpack:

def names_of(lst):
    out = []
    for x in (lst or []):
        if isinstance(x, dict): n = x.get("name") or x.get("tagName") or ""; n and out.append(n)
        elif isinstance(x, str): out.append(x)
    return out

article SSR path

initialState.articleDetail.articleDetailData.data.widgetContent   ← 全文 HTML
                                              .widgetTitle
                                              .publishTime
                                              .widgetSummary
                       .articleRecommendData.authorName
                                            .nextItem

newsflash uses the same path, only widgetContent is shorter.

Shared articles use symlinks (140 cross-project)

A single 36Kr article often covers multiple projects (e.g., an industry analysis mentions 5 companies). During scraping, deduplicate using (itemId, itemType) (actually scraped 2530), and save under the first referencing project's articles/ directory.

Phase 3b scans all articles_index.json and creates relative symlinks for other referencing projects:

target = events/<其他pid>/articles/<itemId>-p.md
src = events/<canonical pid>/articles/<itemId>-p.md
target.symlink_to(Path("..") / ".." / canonical_pid / "articles" / fname)

git tracks symlinks fine (it only stores the target path, does not copy content).

Image deduplication + markdown rewriting

Name each image by its URL sha1 (extension inferred from the URL format,jpg), 9590 unique URLs → 9448 successful.

Phase 5 rewriting:

md_text = md_text.replace(f"![]({url})", f"![](../../../_images/{sha1}.{ext})")

Note:Do not rewrite during the inbox stage(directories still need to be mv'd), after rewriting the path becomes locked. In this case phase 5 ran before mv, and after rewriting the ../../../_images/ paths happened to also be correct under sources (because events/<pid>/articles/ → events → root → _images, 3 levels up).

Anti-patterns

• ❌ Using Playwright for every article: 5–10 times slower; in reality only the list page needs Playwright + cookie warmup
• ❌ Guessing captcha bypasses: wastes time; first try www. / m. / different UAs / Googlebot, 90% chance one of them is open
• ❌ Using WebFetch to scrape SPAs: WebFetch does not run JS, SPA list pages will definitely be empty, skip directly
• ❌ Manually maintaining cookie strings: cookies have short TTLs and expire after 1 hour of running; Playwright credentials:'include' automatically renews them
• ❌ Modifying packets without looking at API error codes:partner_id 不可以为空 ≠ NPE; the former means missing wrapper, the latter means missing cookie/state, the fixes are completely different
• ❌ Judging field types based on a single detail page: the same field across different projects may be nested or a bare string (encountered with industryList, currentFinancing); when writing rewriting code, first use polymorphic unpacking like names_of()

Security / compliance boundaries

• 36Kr pitchhub data is publicly accessible, no login wall
• This project uses a local proxy (clash 7890), no dedicated crawler IPs, simulates normal browser pacing (concurrency 8–16)
• Scraping frequency is gentle (total runtime ~30 minutes to finish 1.5 GB), did not trigger any additional rate limiting
• Purpose: long-term personal knowledge base archive, not distributed to third parties, not for commercial use

Rerun / maintenance

See ~/kb/sources/reference/36kr-pitchhub-2026/_index.md "Rerun pipeline" section for details.

When scraping 2027 data next time, only modify financingTimeList:["2027"] + retest industry totals (new industries may be added).

---

2026-05-19 actual test: 1 hour of exploration + 30 minutes to run the complete pipeline, 1.5 GB of data.

2026-09-26 addendum: The success rate of the 2025 batch must be verified against content

This run only organized existing materials and did not perform new online scraping. The 2025 directory contains 2973 list records and 2514 project IDs, but only 42 project summaries; among 135 article paths, 39 newsflashes are actually empty, and the 96 full-text paths contain one cross-directory duplicate.phase3_failed=0 Therefore full-text completeness cannot be proven. The gap of 29 between the metadata target of 3002 and the actual 2973 also cannot be treated as equivalent to the 29 located missing events.

Acceptance requires separately recording list coverage, detail coverage, non-empty full text, deduplicated full text, images, and external link read status. Failure logs include retries and should be aggregated by entity ID; full text must be checked for title and body, not just HTTP success or file existence. When comparing across batches, normalize image addresses before comparing body text to avoid misjudging articles as new due to absolute paths being changed to relative paths.

Related reports may belong to investors, invested companies, or a single mention in a weekly report. The directory year only represents the funding list filter, not the article year; list counts certainly do not represent full-text article counts. 53 new unique Markdown files have been read in full batches and digests written; image tables within them have not yet been recognized line by line, so it cannot be claimed that the corresponding image details have been digested. See also 2025 Evidence Boundaries and Cases and the original list (local reference material).

Old archives must also have their digestion status verified

In this round, 42 existing source copies were re-read, and together with 53 new unique full texts, a total of 95 per-article digests were produced. Existing files do not prove entry into the wiki; checks must separately record "archive exists", "full text read", and "mechanism entered wiki". The 96 full-text paths contain one duplicate of the same article, so counts should be deduplicated. Full-text coverage here refers only to saved Markdown; images and external links still need separate accounting.

Related directories also do not guarantee matching of subject, year, and article type. The 2025 list can link to reports, recruitment posts, and stock market evening papers from 2020–2026; funding amount, valuation, contract amount, recognized revenue, and ARR cannot share the same amount column. Shell entries and indexes maintain their corresponding status and are not marked as fully digested by writing a summary. See 2025 List and Related Cases.

Sources and related materials

• China Venture Capital 2026 H1 Panoramic Snapshot
• Wang Kaiduo's Agent Browser Matrix Architecture (v1.1 Practical Guide)