Sitemap schedules

An ordinary schedule captures one address. A sitemap schedule points at a sitemap instead — often https://example.com/sitemap.xml — and at each run captures the pages it lists, with one Web Profile and one cadence for all of them.

A page schedule takes at most 10 addresses in one go, each becoming a schedule of its own. To get a sitemap's addresses as a list instead, use Extract URLs.

1. 1

   Choose Sitemap

   In the new schedule form, the first choice is Pages or Sitemap. Choose Sitemap. Without a plan that includes it, the choice is locked and marked Pro.
2. 2

   Enter the sitemap address

   An XML sitemap, a sitemap index, or a text file of URLs. Only pages on the sitemap’s own site are captured.
3. 3

   Narrow it down, if you want

   Fill in Only pages matching, Leave out pages matching and Pages per run. All three are optional.
4. 4

   Preview the pages

   Reads the sitemap now with these settings — the same decision every run makes — and shows how many pages will be captured each run, about how many a month at the chosen frequency and how that compares with your quota, and the pages the run takes.
5. 5

   Finish the rest of the form

   Choose the Web Profile and when it runs, as for any schedule, and save. The Web Profile’s location must be one with dedicated proxies — the form lists them — and the schedule runs at most once a day.

A pattern is matched against the page’s path and query, ignoring case, and * stands for anything:

• Every pattern starts with / or *.
• At most 20 patterns in each box, and at most 200 characters each.

1. The sitemap is read again.
2. Pages on other sites are dropped. The sitemap’s own site includes its subdomains, and www. makes no difference.
3. Only pages matching is applied, then Leave out pages matching.
4. Up to Pages per run are taken, least recently captured first. With 2,400 matching pages at 1,000 a run, each page is captured every 3 runs; the preview and each run say so.
5. The pages are requested one after another, spaced to the capacity of the location’s dedicated proxies and never faster than your plan’s rate limit. With three proxies that is about one page every 10 seconds, so 1,000 pages take about 2¾ hours.

A sitemap schedule runs on its schedule only: Quick Capture and the Auto Schedule API’s trigger are not available for it. To run one now, create it with Start set to Immediately.

A sitemap schedule’s page has a Sitemap runs panel, with one row per run: when it ran, that the schedule started it, and what happened.

The pages themselves appear among the schedule’s captures like any other, each under its own address.

Extract URLs, on the Auto Schedules page, reads a sitemap and downloads its pages as a CSV of address and last-modified date. It creates nothing, and it is available on every plan, the free trial included. It says how many pages the sitemap lists and which child sitemaps could not be read.

One sitemap every two minutes per workspace, because each one makes us fetch the site’s files. A mistyped address or a file that is not a sitemap can be tried again at once.

• The schedule counts as one schedule towards your plan’s limit, however many pages it has.
• Every page captured is one capture from your quota.
• If the quota or the storage runs out partway through, the run stops at that page and says so. The pages after it are not captured in that run.
• A run that stops early because the site is failing gives back the pages it did not try.

• An XML sitemap listing pages.
• A sitemap index, one level deep: up to 50 child sitemaps on the same site are read. The preview says how many were read.
• A plain text file with one URL per line.
• Any of these compressed with gzip.

Up to 50,000 addresses in all, and 20 MB per file, are read. When a sitemap is larger, runs use its first part, and the preview says so.

• A sitemap schedule runs at most once a day, on every plan.
• Its Web Profile’s location must have dedicated proxies; the form and the error name the current ones. A schedule whose profile is later moved elsewhere fails its runs, charging nothing, until it is moved back.