Crawling and indexing

To index pages for related links selection, Graphite’s bot, user agent: GraphiteBot/1.0 (+https://www.graphitehq.com), crawls pages in the sitemap provided.

The daily crawling run starts at 00:00 UTC and crawls 60-240 pages per minute.

During the initial submission of the configuration the crawl will be queued for crawling. Endpoints will be provided by your Graphite POC. The endpoints will not produce related links results until the crawl and indexation is complete.

Once the initial crawl is complete we will re-index the sitemap once per week, related links for each page will be updated to look for more closely related links upon the successful completion of each crawl and will be added to the response.

Info about our crawler

Diagnostic reporting IP: 3.21.78.26
Internal Links API IP: 3.131.206.24

Graphite user agent

Graphite’s bot, user agent: GraphiteBot/1.0 (+https://www.graphitehq.com)