DeepScrape: reusable CSS selectors for cheaper agent crawling, plus an MCP server

Prasenjit Sarkar @stretchcloudOriginally posted 2026.10.23

Open original on X

Carl Hendys team at Salience just published real server log data from 1.5 billion requests. The headline nobody is using yet: distinct sources accessing websites through Claude Code grew about 26x between June and September, mostly hitting technical documentation. Meanwhile OpenAIs own crawler, OAI-SearchBot, more than doubled its requests over the same window while live ChatGPT-User fetches during a conversation fell 80%. Agents are moving from fetching a page on demand to crawling and indexing ahead of time. That shift changes what a scraper needs to be good at. A live fetch inside a chat just needs to survive once. An agent that is going to hit the same documentation site weekly, or crawl a catalog nightly to keep an index fresh, needs extraction that does not re-read the raw HTML with an LLM on every single pass. That, in my experience, is the expensive part most scraping tools never fix: Firecrawl and similar tools still lean on inference per fetch, fine for one page and brutal at the scale this data describes. DeepScrape generates a CSS selector once with an LLM, then reuses it deterministically until the page actually changes, at which point it self-heals instead of burning a model call on every request. The LLM only does targeted work: generating the selector, validating it, adapting it when the DOM shifts. Add /api/map for URL discovery, an MCP server so Claude and Cursor can call the scraper as a tool, and Site-to-MCP to turn any site into a queryable data source, and the whole thing is built for exactly the crawl-ahead pattern Hendys logs are showing. If your agents are about to start crawling instead of fetching, the extraction layer is where the bill actually lands. https://github.com/stretchcloud/deepscrape https://x.com/carlhendy/status/2113484159276695701

This post quotes another post: x.com/carlhendy/status/2113484159276695701