The X-Robots-Tag response header defines how crawlers should index URLs.
While not part of any specification, it is a de-facto standard method for communicating with search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header to adjust how to present web pages or other resources in search results....defines how crawlers should index URLs. While not part of any...resources in search results. Indexing rules defined via <meta name="robots">...