636 servers · Data & APIs · Web Scraping & Crawling
Web Scraping & Crawling servers from Data & APIs, aggregated from every major registry, trust-ranked, and ready to install.
Access a versatile collection of tools for media generation, web scraping, and cryptocurrency research. Convert between data formats, summarize long documents, and extract text from images or PDFs with ease. Streamline your workflow by automating tasks like SEO metadata extraction, code reviews, and content creation.
Enables web browsing capabilities through tools for content extraction, link following, and browser automation with customizable parameters for scraping, data collection, and web crawling tasks.
Integrates with the Serper API to enable web searches and webpage content extraction, supporting research, content aggregation, and data mining tasks.
Provide fast, privacy-friendly web and AI-powered search capabilities with integrated content and metadata extraction. Enhance your AI assistants by enabling comprehensive web scraping without requiring API keys. Optimize performance with caching and secure usage through rate limiting and user agent rotation.
Extracts structured data from web pages based on natural language descriptions, converting website content into JSON format without custom scraping code.
Integrates with Search1API to enable web searches, website crawling, and content extraction with configurable parameters, featuring specialized tools for image search and integration with Coze AI for generating structured research plans.
B2B lead generation tool: search Google Maps by 'plumbers in Austin', get back business profiles with emails, phone numbers, ratings, websites. The differentiator over plain Google Maps API is the contact-enrichment layer (scraped from each business's site). Built for agency prospecting workflows.
Bring sports data into an AI workflow without writing four ESPN scrapers. Useful for fantasy-sports copilots, sports-betting research agents, content sites that auto-summarize game results, or hobby projects where you want Claude to talk to you about last night's game.
Provides specialized investment research tools for analyzing SEC filings, earnings calls, financial data, stock market information, private company details, funding rounds, M&A transactions, and web scraping capabilities.
58-endpoint utility hub - DNS lookup, web scraping, CVE scanning, AI translation, PII detection, domain health audit, company intelligence. Pay per call via x402 on Base.
Analyze web pages and entire sites for SEO, accessibility, performance, and security compliance. Generate detailed reports and actionable issue lists to improve site quality and user experience. Track historical audit data to monitor improvements across multiple crawl sessions.
Licensed creator content for AI agents. Discover music, video, and text catalogs as pre-indexed Pockets — a few hundred tokens instead of a 10,000-token scrape, delivered in under 30ms. Free discovery; pulls are licensed per-use via HTTP 402 paywall — connection is the contract, and 85% of every pull pays the rights holder instantly. **Tools:** `list_pockets` · `search_pockets` · `pull_content` No key needed to browse. Pulling content returns a 402 with signup instructions — register at the URL provided, attach your API key as a Bearer token, and pull licensed content with full attribution and audit trail.
Licensed creator content for AI agents. Discover music, video, and text catalogs as pre-indexed Pockets — a few hundred tokens instead of a 10,000-token scrape, delivered in under 30ms. Free discovery; pulls are licensed per-use via HTTP 402 paywall — connection is the contract, and 85% of every pull pays the rights holder instantly. **Tools:** `list_pockets` · `search_pockets` · `pull_content` No key needed to browse. Pulling content returns a 402 with signup instructions — register at the URL provided, attach your API key as a Bearer token, and pull licensed content with full attribution and audit trail.
Integrates with Octagon API to provide multi-source data aggregation, web scraping, academic research synthesis, competitive analysis, market intelligence, technical analysis, policy research, and trend analysis for professional-grade research capabilities.
▎ Tokyo housing market data for foreigners. Search 18,000+ weekly-scraped rental listings, compare Tokyo's 23 wards, estimate move-in costs, and get data-backed negotiation strategies. Data sourced from SUUMO, Homes.co.jp, and at Home. ▎ 7 tools: search_listings, get_neighborhood_info, estimate_move_in_cost, get_buy_vs_rent, ward_matchmaker, lease_renewal_analyzer, negotiation_intelligence. ▎ Free, no authentication required.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Extracts LinkedIn profile data, company information, and connection details using advanced anti-detection web scraping techniques for recruitment automation, lead generation, and professional network analysis.
Provides a bridge to Dumpling AI's data extraction API for performing web searches, scraping content, extracting structured data, and processing various document formats through 20+ specialized tools.
Enables comprehensive web research by leveraging Tavily's Search and Crawl APIs to aggregate information from multiple sources, extract detailed content, and structure data specifically for generating technical documentation and research reports.
Integrates with Olostep's web scraping API to extract webpage content in markdown format, discover website URLs through search queries, and retrieve structured Google search results with country-specific routing and JavaScript rendering support.
Integrates with Bing Webmaster Tools API to provide website management and SEO analytics through over 40 specialized tools for site management, traffic analysis, crawling diagnostics, URL submission, sitemap management, keyword research, and link analysis.
Integrates with the Scrapezy API to extract structured data from websites based on user-specified prompts, enabling flexible web scraping for data collection, content aggregation, and automated research tasks.
Integrates with OLX marketplaces across five European domains using web scraping to search listings with filters and retrieve detailed property information including seller data for market research and price monitoring.
Provides web search, page extraction, crawling, scraping, and research tools for AI agents through a hosted or self-run MCP server.
Combines Brave Search with web scraping to provide deep research capabilities by extracting full content from pages and traversing links at configurable depths
Website cloning engine with intelligent crawling, asset downloading, PDF generation, authentication support, and dynamic content rendering for website archival, offline browsing, and data extraction workflows
Converts URLs to Markdown, scrapes web content, extracts links, and generates summaries of web pages for efficient content analysis and data extraction.
Provides conversational access to Palo Alto Networks Cortex Cloud platform documentation through web scraping and intelligent indexing with automatic caching, relevance scoring, and separate tools for general documentation versus API-specific content.
Extract clean markdown from any URL. Removes boilerplate. For RAG pipelines. x402.
Turn any website or API into structured JSON with LLM-authored declarative scraping configs.
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
Unified data server: stocks, crypto, news, weather, web scraping, FX rates in one MCP
Web-scraping toolkit with 22 tools for structured web data as JSON for AI agents.
Turn any website or API into structured JSON with LLM-authored declarative scraping configs.
One MCP for 160+ live web-data APIs — clean JSON from sites that block scrapers.
Post-scrape data cleaner, no LLM: repairs mojibake, HTML, invisible chars. Plus a verdict.
AI-agent marketplace: agent templates, web search & crawl, SEO audit, RO company data, city info.
Web data for AI agents: scrape, crawl, search, deep research, site monitoring, browser automation
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
Web toolkit for AI agents: read any URL as Markdown, screenshot, extract data, or crawl a site.
SERP data scraper: Scrape Search Results and get all the data you need. On the JoJ API marketplace.
Crawlbase MCP — wraps the Crawlbase Crawling API (crawlbase.com, formerly
One API for public web data across social, directories and real estate, as clean JSON.
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
Ranks the best AI tool or API per task: transcription, TTS, web search, scraping and OCR.
Cloudflare Solver: Scraping API designed to bypass Cloudflare protection.
Made-to-order data for AI agents: company intel, B2B contacts, scraping. Pay per call via x402.
Fetch and process content from specified URLs & sources using the Oxylabs Web Scraper API.
Unofficial TikTok API & scraper: creator analytics, video data, comments, search. x402, no API key.
FAQ
636 Web Scraping & Crawling MCP servers are indexed on Lulu MCPs, ranked by trust score and aggregated from the official MCP Registry, Glama, PulseMCP and Smithery. Every listing links back to its source registry and installs in one click for Claude Code, Cursor and VS Code.
Web Scraping & Crawling is a sub-filter within Lulu MCPs' Data & APIs category, matched against a validated term set specific to web scraping & crawling. Servers can appear under more than one sub-filter when their functionality spans multiple areas.
