The X-Robots-Tag response header defines how crawlers should index URLs.
While not part of any specification, it is a de-facto standard method for communicating with search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header to adjust how to present web pages or other resources in search results....Use a maximum of <number> characters as a textual snippet for...