Crawling and indexing

To index pages for related links selection, Graphite’s bot, user agent: GraphiteBot/1.0 (+https://www.graphitehq.com), crawls pages in the sitemap provided during configuration.

The crawling speed is 240 pages per minute.

During the initial submission of the configuration the crawl will be queued for crawling. Endpoints will be provided in your Configuration page, they will not produce related links results until the crawl and indexation is complete.

Once the initial crawl is complete we will re-index the sitemap once per week, related links for each page will be updated to look for more closely related links upon the successful completion of each crawl and will be added to the response.

If a configuration has been edited, the Configuration will be scheduled to re-crawl your site within 24 hours of the submission of the revised Configuration.

🚧

Checking Module IDs

When a Configuration has been edited, the Module IDs will be updated for any Modules that have been changed to track the changes. You may need to update the Modules IDs in your code as it relates to these revisions.

Info about our crawler

Diagnostic reporting IP: 3.21.78.26
Internal Links API IP: Variable (not static)

Graphite user agent

Graphite’s bot, user agent: GraphiteBot/1.0 (+https://www.graphitehq.com)