The X-Robots-Tag response header defines how crawlers should index URLs.
While not part of any specification, it is a de-facto standard method for communicating with search bots, web crawlers, and similar user agents.
Search-related crawlers use the rules from the X-Robots-Tag header to adjust how to present web pages or other resources in search results.... unavailable_after: <date/time> Requests not to show this...specified <date/time> . Ignored if no valid <date/time> is specified...