Skip to main content
To run SEO crawls around the clock, install Screaming Frog or Sitebulb on a Windows RDP, or Puppeteer and Playwright on a Linux VPS, start the crawl and disconnect. The server keeps crawling while your own computer is off. Long crawls of large sites no longer tie up your laptop or home connection. Every FlashRDP server has a 1 Gbps port and runs 24/7 in Phoenix, AZ or Paris, France. Crawl politely: the Acceptable Use Policy doesn’t allow scraping that ignores robots.txt or a site’s terms, or that overloads the target site.
Crawl responsibly. Respect robots.txt and each site’s terms, limit your request rate, and crawl sites you own or have permission to audit. Don’t rotate proxies or IP addresses to get around a site’s blocks or rate limits. A crawl that overloads a site breaks the AUP and can get your server suspended.

​
Who uses this

Roles: Technical SEOs, data analysts, developers Workflows: Site audits, broken-link checks, monitoring your own or clients’ sites Integrations: Screaming Frog SEO Spider, Sitebulb, Puppeteer, Playwright

​
Running Screaming Frog on Windows

Desktop crawlers such as Screaming Frog SEO Spider and Sitebulb need a Windows desktop, which a Windows RDP provides.
  1. Order a FlashRDP Windows RDP. Large crawls need RAM: GOLD (16 GB) or above suits sites with hundreds of thousands of URLs.
  2. Connect to your server via RDP.
  3. Download Screaming Frog from its official website in the server’s browser and install it.
  4. Set a polite crawl speed under Configuration > Speed (limit threads and URLs per second), and leave Respect robots.txt on.
  5. Start the crawl and close the Remote Desktop window. Don’t sign out: the crawl keeps running while you’re disconnected.
To schedule recurring crawls, see Schedule SEO Audits on Windows.

​
Running Puppeteer or Playwright on Linux

For custom crawlers built with Puppeteer, Playwright or Python’s BeautifulSoup, a headless Linux VPS is enough.
  1. Order a Linux VPS with Ubuntu 24.04.
  2. Log in via SSH.
  3. Install Node.js and npm:
    Ubuntu’s packaged Node.js can lag behind; check that its version meets your library’s minimum with node --version.
  4. Install the system libraries headless Chromium needs. For Playwright, let it install the browser and its dependencies for you:
    For Puppeteer on Ubuntu 24.04:
    On Ubuntu 22.04, drop the t64 suffixes from those package names.
Set a clear user agent that names your crawler and a contact address, add delays between requests, and cache pages you’ve already fetched. Site owners are far less likely to block a crawler they can identify and that doesn’t strain their server.

Schedule SEO Audits on Windows

Secure Your Windows RDP

Connect via SSH

Acceptable Use Policy

Last modified on October 16, 2026