The X-Robots-Tag response header defines how crawlers should index URLs.
While not part of any specification, it is a de-facto standard method for communicating with search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header to adjust how to present web pages or other resources in search results....omitted, search engines may generate a text snippet and video preview...View this page on GitHub • Report a problem with this content...