The X-Robots-Tag response header defines how crawlers should index URLs.
While not part of any specification, it is a de-facTo standard method for communicating with search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header To adjust how To present web pages or other resources in search results....X-Robots-Tag header to adjust how to present web pages or other resources...non-HTML documents like images, PDFs, or other media. Note: Only...