
Capture every page in a sitemap with one schedule
Archiving a whole site one address at a time means a schedule for every page, and a list that is wrong the day someone publishes something new. A sitemap schedule reads the list from the site itself.
· 5 min read
One schedule per page does not scale
A site with 300 pages needs 300 schedules if each one captures a single address. Someone has to create them, and someone has to remember to add the 301st. Most sites already publish the list you need: the sitemap, usually at /sitemap.xml, which their own publishing system keeps up to date for search engines.
How a sitemap schedule works
In the new schedule form, the first choice is Pages or Sitemap. Choose Sitemap, give the sitemap's address, pick a Web Profile and when it runs. From then on, each run does the same four things:
- Reads the sitemap again, so new pages are picked up and removed ones dropped.
- Keeps only pages on the sitemap's own site, subdomains included.
- Applies your patterns, if you set any.
- Captures up to your pages-per-run limit, with the one Web Profile for all of them.
The address can be an XML sitemap, a sitemap index or a plain text file of URLs, compressed with gzip or not. The sitemap schedules guide has the exact limits.
Choosing pages with patterns
You rarely want everything. Two boxes narrow the list: pages to include and pages to leave out. A pattern is matched against the page's path and query, ignoring case, and * stands for anything.
Only pages matching /products/*
/blog/*
Leave out pages matching *?page=*
/blog/tag/*Before saving, Preview the pages shows how many pages match, how many each run will capture, and about how many captures that is in a month compared with your quota.
What a run costs and how long it takes
- Captures. Every page captured is one capture. 1,000 pages every day is about 30,000 captures a month, which is more than the Pro quota of 10,000. Weekly or monthly runs, or a tighter pattern, usually fit better.
- Time. Pages are requested one after another, paced to the capacity of the location's proxies and never faster than your plan's rate limit. At about one page every 10 seconds, 1,000 pages take about 2¾ hours.
- Frequency. A sitemap schedule runs at most once a day, and it runs on its schedule only: there is no Quick Capture for it.
- A failing site stops the run early. Once 20 pages have finished, if more than half failed, the run stops. Pages not yet tried are not captured and not charged.
Where the captures go
Each page appears among the schedule's captures under its own address, and the schedule has a Sitemap runs panel with one row per run. If the profile delivers to Google Drive, Dropbox or an S3 bucket, the pages of one run go into a folder of their own, named by the run's time. With change detection on, each page is compared with its own previous capture, and a webhook can tell your systems when a run has finished.
FAQ
What happens when the site adds a page?
The sitemap is read again at every run, so a page added to it is captured from the next run on, and a page removed from it stops being captured. You do not edit the schedule.
Does a sitemap schedule count as many schedules?
No. It counts as one schedule towards your plan's limit, however many pages it has. Each page captured is one capture from your quota.
What if the sitemap has more pages than one run allows?
The pages take turns, least recently captured first. With 2,400 matching pages and 1,000 pages per run, each page is captured every three runs. The preview and each run say so.
Which plans include sitemap schedules?
Pro, Business and Business Plus, each up to 1,000 pages per run and at most one run a day. Enterprise goes further by arrangement. Extract URLs, which downloads a sitemap as a CSV, is on every plan.