MCPs › Dev tools › Web Crawler MCP Server
An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.
Not monetized yet
Turn Web Crawler MCP Server’s tool calls into revenue: one disclosed sponsored slot, 70% revenue share, fail-open by design.
Install Web Crawler MCP Server
For anyone using Web Crawler MCP Server — no Lulu account needed# stdio server — install per the repository README: https://github.com/dhannusch/smart-web-crawler-mcp
9 field-tested tactics as a designed playbook plus skills your coding agent can run. Free.
Get the Kit →FAQ
Web Crawler MCP Server installs from source — follow the repository README.
Unrated out of 100, computed from cross-registry traction signals (installs, stars, registry presence) — never influenced by sponsorship.
Similar servers
Works well together
Your server?
This is for the person who owns Web Crawler MCP Server — adds Lulu Ads to your own code. Not the install steps above, those are for your users.
Copies a ready prompt: your coding agent installs the lulu-ads SDK, wires the slot, and applies the widget design guide.
or set up manually at getlulu.dev/publishers
Get verified so you can edit the page. Your badge is already live below — no claim needed for that.
Managed hosting with monetization built in — waitlist.
[](https://getlulu.dev/mcps/web-crawler-mcp-server)