Crawlora ORG

Go SDK for the public Crawlora API.

Python SDK for the public Crawlora API.

JavaScript and TypeScript SDK for the public Crawlora API.

Hosted MCP server for structured public web data — 319 tools across search, maps, commerce, social & finance, returning clean JSON.

PHP SDK for the Crawlora web-scraping API

Open aggregate data behind the State of Steam 2026 study — top 10,000 Steam games (public storefront + SteamSpy estimates). CC BY 4.0.

Dead-Web Index — reachability census of the top web (alive/redirect/blocked/dead per probe arm). Data CC BY 4.0.

Is a domain genuinely dead, or just blocking your bot? A passive, local, MIT Go CLI + library that classifies domain reachability (alive/redirect/blocked/dead). The open methodology behind the Crawlora Dead-Web Index.

n8n community node for Crawlora — structured web data from the public web.

No description.

Crawlora Agent Skills — structured public web data (search, e-commerce, social, finance, app stores, media, reviews) via the Crawlora REST API. Installable in any agent with the skills CLI or as a Claude Code plugin marketplace.

Ruby SDK for the Crawlora web-scraping API

Official Java SDK for the Crawlora web-scraping API — typed, dependency-free client with grouped & dynamic operation calls, retries, pagination, hooks, and rate limiting.

Google Trends search-interest data for 12 FIFA World Cup sponsors (2004-2026), correlated against World Cup search interest. CC BY 4.0.

Aggregates over FIFA 22's Steam reviews (130,062 votes, all languages, 2021-2026) and a 217-review current-state sample, read against EA's server shutdown and Steam delisting. CC BY 4.0.

Crawlora integration for OpenClaw: ClawHub MCP skill + native tool plugin.

Open dataset: 39,241 AI Chrome Web Store extensions — install counts, ratings, permissions, first-party vs third-party. CC BY 4.0.

Aggregates over 92,088 App Store + Google Play reviews of the X (Twitter) app, before/after the Musk acquisition. The 4.6-star rating held; written-review sentiment and topics shifted. CC BY 4.0.

Aggregates over 1,610 App Store, Google Play, and Reddit reviews/posts about the Claude app — pricing and rate-limits beat quality complaints in every corpus. CC BY 4.0.

US Housing Affordability Index 2026 — median list price, income, price-to-income ratio, and salary needed to buy for 51 states + 100 cities (CC BY 4.0)

How long hit songs have been, 1959-2025 — every Billboard Year-End Hot 100 hit matched to Spotify (duration + play count). Data behind the Crawlora study. CC BY 4.0.

Crawlora for Zapier — get structured web data and page content from the public web.

Crawlora custom app blueprint for Make.

Open longitudinal host-reachability / dead-web panel from Common Crawl robotstxt (2018-2026), calibrated

2026 World Cup players and nations ranked by TikTok hashtag views plus creator-mention affinity and the co-mention social graph (CC BY 4.0).

World Cup 2026 TikTok creators by nation: median followers, reach, verified and contact rates (CC BY 4.0).

Open robots.txt census of the Tranco top 1M: which AI crawlers (GPTBot, CCBot, ClaudeBot, ...) each site blocks. CC BY 4.0. https://crawlora.net/ai-crawler-index

Open dataset (CC BY 4.0) behind the Crawlora study The Podcast Gold Rush and the Graveyard You Cant See — the discoverable head of Apple Podcasts (US, June 2026): charts + search samples with episode counts, podfade and genre, plus an Apple-vs-Spotify Top-100 overlap.

Open dataset: TikTok engagement rate by niche (4,509 videos, 13 niches). CC BY 4.0. Study: https://crawlora.net/tiktok-engagement-index

Crawlora Streaming Fragmentation Index — where the most-popular US titles stream (popular-100 + a ~5,000-title cross-section). Data CC BY 4.0.

Detect which anti-bot/WAF protects a site and how hard it is to scrape — passive, local, MIT. The open Go CLI + library behind the Crawlora Anti-Bot Adoption Index.

Crawlora Anti-Bot Adoption Index — top-1M open dataset (CC BY 4.0)

n8n community node for the Crawlora Web Scraper — turn any URL into clean Markdown, HTML, links, or metadata. Single-service, AI-agent ready.

This is the official repository for submitting MCP servers to be included in Cline's MCP Marketplace. If you’ve built an MCP server and want it to be discoverable and easily installable by millions of developers using Cline, submit your server here.

A collective list of free APIs

List of libraries, tools and APIs for web scraping and data processing.

A collection of MCP servers.