Access rules for automated agents

We have introduced clear rules for spiders, scrapers, crawlers and other automated agents accessing our hosted websites.

Automated agents must identify themselves using a proper user agent that includes their name and URL, and must not impersonate another service or crawler. They should use the canonical domain, download and follow 'robots.txt' instructions, and respect links marked with 'rel="nofollow"'.

To help protect website performance and avoid unnecessary traffic, automated agents are also expected to rate-limit their requests and avoid repeated downloads.

These guidelines are intended to support responsible, transparent and efficient automated access to our website.

Repeated violations may result in an automated black-listing of your ip address.

Thank you.

CDN outage »