The X-Robots-Tag response header defines how crawlers should Index URLs.
While not part of any specification, it is a de-facto standard method for communicating with Search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header to adjust how to present web pages or other resources in Search results....Skip to main content Skip to search Skip to select language Article...defines how crawlers should index URLs. While not part of any...