Skip to main content
To run SEO audits on a schedule, install Screaming Frog SEO Spider on a Windows RDP, switch it to database storage mode, give it enough memory, and set up a crawl under File > Scheduling (or call its command-line tool from Windows Task Scheduler). The server runs the crawls overnight, even when your own computer is off. Shared web hosting isn’t built for this: it limits CPU, RAM and long-running processes. A laptop has to stay awake and online for the whole crawl. A Windows RDP is a Windows Server desktop you reach over Remote Desktop Protocol (RDP) that runs 24/7.

​
Size the server

Screaming Frog holds crawl data in RAM (memory storage) or on disk (database storage). Large crawls with custom extraction can use many gigabytes. These are starting points: watch memory use in Task Manager during your first crawls and adjust.

​
Set up scheduled audits

​
Prerequisites

  • A FlashRDP Windows RDP with Windows Server 2019, 2022 or 2025
  • Administrator access over Remote Desktop
  • A Screaming Frog license: scheduling and the command-line interface need a paid license. See Screaming Frog for current terms.

​
Steps

  1. Connect: Connect via Remote Desktop and sign in as Administrator.
  2. Install: Download Screaming Frog SEO Spider from its official website and run the installer. It includes its own Java runtime, so you don’t need to install Java separately.
  3. Use database storage: In File > Settings > Storage Mode, choose Database Storage. Crawl data goes to the SSD instead of RAM, which large crawls need.
  4. Set memory: In File > Settings > Memory Allocation, give the crawler most of the server’s RAM while leaving a few GB for Windows (for example 12 GB on GOLD).
  5. Crawl politely: Under Configuration > Speed, limit threads and URLs per second, and keep robots.txt set to respect it. Crawl sites you own or are authorized to audit.
  6. Schedule the crawl: Open File > Scheduling, add a task, choose the start URL, saved configuration, frequency (for example weekly on Sunday at 02:00) and which reports to export.
Prefer Windows Task Scheduler? Point a task at Screaming Frog’s command-line tool, for example:
In the task’s settings, choose Run whether user is logged on or not so it runs while you’re disconnected. Check the install path and flags against Screaming Frog’s user guide for your version.
The Acceptable Use Policy doesn’t allow scraping that ignores robots.txt or a site’s terms, or that overloads the target site. Keep crawl speeds modest, especially for sites behind a CDN or bot protection, and don’t rotate IP addresses to get around blocks.

Host SEO Crawlers

Connect to Your Windows RDP

Secure Your Windows RDP

Last modified on October 16, 2026