crawlee

Crawlee is a web scraping and browser automation library for Node.js. Web scraping means automatically visiting websites and extracting information from them — like prices, product listings, article text, or any other data you can see in a browser. Crawlee makes this easier by handling the repetitive, technical work for you.

The problem it solves is that scraping modern websites is hard: pages load content using JavaScript, websites detect and block automated requests, and managing a queue of thousands of URLs while handling errors and retries gets complex fast. Crawlee handles all of this. It can control real browsers (via Playwright or Puppeteer) to scrape JavaScript-heavy sites, or use fast HTTP requests for simpler pages. It automatically rotates proxies to avoid blocks, generates realistic browser fingerprints to appear human-like, manages a queue of URLs to visit, and saves collected data to disk or cloud storage.

You would use this if you need to extract data from websites at scale — for example, to build a price comparison tool, aggregate news articles, collect training data for AI, or monitor competitor websites. It works in JavaScript and TypeScript and runs on Node.js. It is developed by Apify, a company that provides cloud infrastructure for running scrapers, though Crawlee itself runs anywhere.